Image decoding apparatus, image transmission apparatus, image processing system, image decoding method, and image transmission method using range information

ABSTRACT

An image decoding apparatus includes a request unit that requests an image transmission apparatus to transmit an image, a transmission unit that transmits range information indicating a range which is a partial range of the image and which is selected in accordance with a user operation to the image transmission apparatus, a receiving unit that, while the partial range of the image is selected, receives, from the image transmission apparatus, the entirety of the image when the image belongs to a first type and a portion of the image when the image belongs to a second type, by receiving one or more tiles, which corresponds to the partial range of the image and outside which a reference to is prohibited in inter-picture prediction, among a plurality of tiles included in the image, and a decoding unit that decodes the received entirety or portion of the image.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/933,520, filed on Jan. 30, 2014, the contents of which are herebyincorporated by reference.

BACKGROUND

1. Technical Field

The present disclosure relates to an image decoding apparatus and thelike that receive an encoded image from an image transmission apparatusand that decode the image received from the image transmissionapparatus.

2. Description of the Related Art

The High Efficiency Video Coding (HEVC) scheme is presently beingexamined as a new image codec standard (see JCTVC-N1005, “HighEfficiency Video Coding (HEVC) Range extension text specification draft4”). The HEVC scheme enables transmission and reception of high-qualityimages with low bit rates, compared to existing schemes. It isanticipated that transmission and reception of data, including images,will be carried out more actively in future.

There have recently been an increasing number of services for theprocessing of data including images, in which information related toimages, such as subtitles, broadcast program information,advertisements, or commercial breaks, or information used to improveuser convenience is attached to the images. In such services,unnecessary information may be provided.

SUMMARY

One non-limiting and exemplary embodiment provides an image decodingapparatus and the like for efficient processing of an encoded image.

Additional benefits and advantages of the disclosed embodiments will beapparent from the specification and Figures. The benefits and/oradvantages may be individually provided by the various embodiments andfeatures of the specification and drawings disclosure, and need not allbe provided in order to obtain one or more of the same.

In one general aspect, the techniques disclosed here feature an imagedecoding apparatus according to an aspect of the present disclosure isan image decoding apparatus for receiving an encoded image from an imagetransmission apparatus and decoding the image received from the imagetransmission apparatus. The image decoding apparatus includes a requestunit that requests the image transmission apparatus to transmit theimage, a transmission unit that transmits range information to the imagetransmission apparatus, the range information being range informationindicating a range that is a partial range of the image within an entirerange of the image and that is selected in accordance with a useroperation, a receiving unit that, while the partial range of the imageis selected, (i) receives the entirety of the image from the imagetransmission apparatus when the image belongs to a first type, and (ii)receives a portion of the image from the image transmission apparatuswhen the image belongs to a second type, by receiving one or more tilescorresponding to the partial range of the image among a plurality oftiles included in the image, each of the one or more tiles being a tileoutside which a reference to is prohibited in inter-picture prediction,and a decoding unit that decodes the entirety or portion of the imagereceived by the receiving unit from the image transmission apparatus.

These general and specific aspects may be implemented using a system, amethod, and a computer program, and any combination of systems, methods,and computer programs.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a reference relationship of a blockbetween tiles;

FIG. 2 is a diagram illustrating a reference relationship of a blockwithin a constrained tile;

FIG. 3 is a block diagram illustrating an example configuration of imageprocessing system according to a first embodiment;

FIG. 4 is a block diagram illustrating an example configuration of imagedecoding apparatus according to the first embodiment;

FIG. 5 is a diagram illustrating an example of the syntax that specifiesa constraint for motion compensation according to the first embodiment;

FIG. 6 is a diagram illustrating an example of a bit stream includingthe syntax that specifies a constraint for motion compensation accordingto the first embodiment;

FIG. 7 is a diagram illustrating an example of the processing flow ofthe image processing system according to the first embodiment;

FIG. 8 is a diagram illustrating an example of an image data selectionprocess according to the first embodiment;

FIG. 9 is a diagram illustrating an example of first allocation of amain image and sub-information to a plurality of tiles in an imageaccording to the first embodiment;

FIG. 10 is a diagram illustrating an example of second allocation of amain image and sub-information to a plurality of tiles in an imageaccording to the first embodiment;

FIG. 11 is a diagram illustrating an example of third allocation of amain image and sub-information to a plurality of tiles in an imageaccording to the first embodiment;

FIG. 12 is a diagram illustrating an example of fourth allocation of amain image and sub-information to a plurality of tiles in an imageaccording to the first embodiment;

FIG. 13 is a diagram illustrating an example of the timing at which theallocation of a main image and sub-information to a plurality of tilesin an image according to the first embodiment is changed;

FIG. 14 is a diagram illustrating an example of an operation of changingthe allocation of a main image and sub-information to a plurality oftiles in an image according to the first embodiment;

FIG. 15 is a diagram illustrating an example of the processing flow forchanging the allocation of a main image and sub-information to aplurality of tiles in an image according to the first embodiment;

FIG. 16 is a diagram illustrating an example of an operation ofscrambling some of a plurality of tiles in an image according to thefirst embodiment;

FIG. 17 is a block diagram illustrating an example configuration of aprocessing unit of a server according to the first embodiment;

FIG. 18 is a diagram illustrating an example display in a first modeaccording to a second embodiment;

FIG. 19 is a diagram illustrating an example display in a second modeaccording to the second embodiment;

FIG. 20 is a diagram illustrating an example display in a third modeaccording to the second embodiment;

FIG. 21 is a schematic diagram illustrating an example configuration ofa broadcast system according to a fourth embodiment;

FIG. 22 is a block diagram illustrating an example configuration of atelevision according to the fourth embodiment;

FIG. 23 is a diagram illustrating an example of the processing flow ofthe broadcast system according to the fourth embodiment;

FIG. 24 is a diagram illustrating an example of the processing flow ofan image processing system according to a fifth embodiment;

FIG. 25 is a diagram illustrating the example (continued) of theprocessing flow of image processing system according to the fifthembodiment;

FIG. 26 is a diagram illustrating an example of the timing at whichdisplay of the entirety of an image and display of a portion of theimage are switched according to the fifth embodiment;

FIG. 27 is a diagram illustrating an example of partial displayaccording to the fifth embodiment;

FIG. 28 is a diagram illustrating an example of entire display accordingto the fifth embodiment;

FIG. 29 is a block diagram illustrating an example characteristicconfiguration according to the fifth embodiment;

FIG. 30 is a diagram illustrating an example characteristic operation ofthe processing flow according to the fifth embodiment;

FIG. 31 is an overall configuration diagram of a content supply systemthat achieves the provision of a content distribution service;

FIG. 32 is an overall configuration diagram of a digital broadcastingsystem;

FIG. 33 is a block diagram illustrating an example configuration of atelevision;

FIG. 34 is a block diagram illustrating an example configuration of aninformation reproducing/recording unit that reads and writes informationfrom and to a recording medium serving as an optical disk;

FIG. 35 is a diagram illustrating an example structure of the recordingmedium serving as an optical disk;

FIG. 36A is a diagram illustrating an example of a mobile phone;

FIG. 36B is a block diagram illustrating an example configuration of themobile phone;

FIG. 37 is a diagram illustrating the configuration of multiplexed data;

FIG. 38 is a diagram schematically illustrating how each stream ismultiplexed in multiplexed data;

FIG. 39 is a diagram illustrating in more detail how a video stream isstored in a PES packet sequence;

FIG. 40 is a diagram illustrating the structure of TS packets and sourcepackets in multiplexed data;

FIG. 41 is a diagram illustrating the data configuration of a programmap table (PMT);

FIG. 42 is a diagram illustrating the internal configuration ofmultiplexed data information;

FIG. 43 is a diagram illustrating the internal configuration of streamattribute information;

FIG. 44 is a diagram illustrating steps for identifying video data;

FIG. 45 is a block diagram illustrating an example configuration of anintegrated circuit that implements a moving image encoding method and amoving image decoding method in each embodiment;

FIG. 46 is a diagram illustrating a configuration for switching adriving frequency;

FIG. 47 is a diagram illustrating steps for identifying video data andswitching a driving frequency;

FIG. 48 is a diagram illustrating an example of a look-up table in whichvideo data standards and driving frequencies are associated with eachother;

FIG. 49A is a diagram illustrating an example of the configuration of asignal processing unit in which a module is shared; and

FIG. 49B is a diagram illustrating another example of the configurationof a signal processing unit in which a module is shared.

DETAILED DESCRIPTION

Underlying Knowledge of the Present Disclosure

The present inventors have encountered problems with an imagetransmission apparatus, an image decoding apparatus, and the like. Thedetails will be described hereinafter.

In the International Telecommunication Union TelecommunicationStandardization Sector (ITU-T) standard, referred to as H.26x, or in theInternational Standards Organization/International ElectrotechnicalCommission (ISO/IEC) standard, referred to as MPEG-x, as illustrated inFIG. 1, a picture is divided into a plurality of rectangular unitscalled tiles. Each tile is not capable of reference to other tile areaswithin the screen (or within the picture). For example, if a second tileis a tile to be encoded or decoded, the second tile is not capable ofreference to a first tile.

Note that in the case of temporally different image signals, a currentblock in the tile to be encoded or decoded is capable of reference to anarea different from the tile to be encoded or decoded. For example, inFIG. 1, an m-th picture corresponds to an image signal at time t−1, andan n-th picture corresponds to an image signal at time t. These imagesignals are temporally different image signals. In this case, a block Cto be encoded or decoded in the tile to be encoded or decoded is capableof reference to an area different from the area of the second tile.

As used herein, the phrase “a given tile, block, or the like has areference to another tile, block, or the like” means that another tile,block, or the like is referenced in order to process a given tile,block, or the like. FIG. 1 illustrates an example in which the block Cincluded in the second tile in the n-th picture at the time t isprocessed by using a reference area R included in a third tile in them-th picture at the time t−1 as a reference image.

However, if the above-described reference is allowed, the followingissue occurs. For example, in a technique, when the screen size (orpicture size) is very large, a plurality of tile areas are processedusing separate large-scale integration (LSI) devices in parallel.However, as described above, if a reference to an image signal that isdifferent in time is allowed, it may be necessary to write image signalsgenerated using separate LSI devices to a single large memory and toshare the memory. In this case, it may be necessary to synchronize theseparate LSI devices with each other. In addition, the amount ofprocessing for synchronization may increase.

JCTVC-N1005, “High Efficiency Video Coding (HEVC) Range extension textspecification draft 4”, noted above, describes a method by whichinformation indicating a constrained reference to an image signal thatis different in time is included in a coded stream. In this case, asillustrated in FIG. 2, the block C to be processed has a reference to areference area R included in the corresponding tile area (i.e., thesecond tile) in a temporally different image signal (e.g., the m-thpicture). Accordingly, in a case where a plurality of LSI devicesperform parallel processing for a plurality of tile areas, not allreference images may be output to the shared memory.

In the following, the above-described tile is referred to as an“MV-constrained tile” or a “constrained tile”. In addition, an area thatcan be referenced is not limited to only the corresponding tile area.For example, a reference area R may be specified in an area includingthe first tile and the second tile in the m-th picture.

By using the MV-constrained tile described above, it is possible toencode or decode an area that is a portion of an image without anyreference to other areas.

Accordingly, the inventors have contemplated the possibility of an imagebeing transmitted, received, and decoded on a tile-by-tile basis.

For example, an image decoding apparatus according to an aspect of thepresent disclosure is an image decoding apparatus for receiving anencoded image from an image transmission apparatus and decoding theimage received from the image transmission apparatus. The image decodingapparatus includes a request unit that requests the image transmissionapparatus to transmit the image; a transmission unit that transmitsrange information to the image transmission apparatus, the rangeinformation being range information indicating a range which is apartial range of the image within an entire range of the image and whichis selected in accordance with a user operation; a receiving unit that,while the partial range of the image is selected, (i) receives theentirety of the image from the image transmission apparatus when theimage belongs to a first type, and (ii) receives a portion of the imagefrom the image transmission apparatus when the image belongs to a secondtype, by receiving one or more tiles corresponding to the partial rangeof the image among a plurality of tiles included in the image, each ofthe one or more tiles being a tile outside which a reference to isprohibited in inter-picture prediction; and a decoding unit that decodesthe entirety or portion of the image received by the receiving unit fromthe image transmission apparatus.

The configuration described above allows the image decoding apparatus toswitch between decoding of the entirety of the image and decoding of aportion of the image in accordance with the type of the image. Thus, theimage decoding apparatus enables the efficient processing of an encodedimage.

For example, the image decoding apparatus may further include a displaycontrol unit that displays the entirety or portion of the image decodedby the decoding unit on a first display device.

The configuration described above allows the image decoding apparatus toswitch between display of the entirety of the image and display of aportion of the image.

For example, the range information may be saved. In a case where thereceiving unit receives a portion of the image, the receiving unit mayreceive a portion of the image in accordance with the saved rangeinformation. In a case where the decoding unit decodes a portion of theimage, the decoding unit may decode the portion of the image received inaccordance with the saved range information. In a case where the displaycontrol unit displays a portion of the image, the display control unitmay display the portion of the image received and decoded in accordancewith the saved range information.

The configuration described above allows the image decoding apparatus toreceive, decode, and display a portion of the image in accordance withthe range information saved in the image decoding apparatus, the imagetransmission apparatus, or the like.

For example, in a case where the display control unit displays a portionof the image after displaying the entirety of another image, the displaycontrol unit may display the portion of the image received and decodedin accordance with the saved range information.

The configuration described above allows the image decoding apparatus toswitch display of the image from display of the entirety of anotherimage to display of a portion of the image.

For example, in a case where the display control unit further stopsdisplay of a portion of the image and then resumes the display of theportion of the image, the display control unit may display the portionof the image received and decoded in accordance with the saved rangeinformation.

The configuration described above allows the image decoding apparatus toresume the display of the portion of the image in accordance with therange information saved in the image decoding apparatus, the imagetransmission apparatus, or the like.

For example, the display control unit may further display the portion ofthe image received and decoded in accordance with the saved rangeinformation on a second display device different from the first displaydevice.

The configuration described above allows the image decoding apparatus tocause a display device to inherit a display format from another displaydevice.

For example, in a case where the range information is not saved, thetransmission unit may transmit the range information to the imagetransmission apparatus. In a case where the range information is savedand in a case where the receiving unit receives a portion of the image,the receiving unit may receive a portion of the image in accordance withthe saved range information without the range information beingtransmitted from the transmission unit to the image transmissionapparatus.

The configuration described above allows the image decoding apparatus toreceive, decode, and display a portion of the image in accordance with,for example, saved range information transmitted from another imagedecoding apparatus.

For example, the image decoding apparatus may further include aselection unit that selects a partial range of the image in accordancewith the user operation. The transmission unit may transmit rangeinformation indicating the range selected by the selection unit to theimage transmission apparatus.

The configuration described above allows the image decoding apparatus toappropriately select a portion to be received and decoded, in accordancewith a user operation.

For example, while the partial range of the image is selected, thereceiving unit may receive the entirety of the image when the imagebelongs to the first type, the first type being a type in which displayof the entirety of the image is recommended, and the receiving unit mayreceive a portion of the image when the image belongs to the secondtype, the second type being a type in which display of the entirety ofthe image is not recommended.

The configuration described above allows the image decoding apparatus toreceive the entirety of the image when display of the entirety of theimage is recommended, to receive a portion of the image when display ofthe entirety of the image is not recommended. Thus, the image decodingapparatus enables reception and decoding of an appropriate image.

For example, while the partial range of the image is selected, thereceiving unit may receive the entirety of the image when the imagebelongs to the first type, the first type being an advertisement, andthe receiving unit may receive a portion of the image when the imagebelongs to the second type, the second type being different from anadvertisement.

The configuration described above allows the image decoding apparatus toreceive the entirety of the image for an advertisement for which theentirety of the image is desirably displayed, and to receive a portionof the image for a main part of a broadcast program for which theentirety of the image may not necessarily be displayed.

For example, the image may be assigned a flag indicating whether theimage belongs to the first type or the second type, and the receivingunit may receive the entirety or portion of the image in accordance withthe flag.

The configuration described above allows the image decoding apparatus toreceive the entirety or portion of the image in accordance with the flagassigned to the image.

For example, an image transmission apparatus according to an aspect ofthe present disclosure is an image transmission apparatus fortransmitting an encoded image to an image decoding apparatus. The imagetransmission apparatus includes a request acceptance unit that accepts arequest from the image decoding apparatus to transmit the image; areceiving unit that receives range information from the image decodingapparatus, the range information being range information indicating arange which is a partial range of the image within an entire range ofthe image and which is selected in accordance with a user operation; anda transmission unit that, while the partial range of the image isselected, (i) transmits the entirety of the image to the image decodingapparatus when the image belongs to a first type, and (ii) transmits aportion of the image to the image decoding apparatus when the imagebelongs to a second type, by transmitting one or more tilescorresponding to the partial range of the image among a plurality oftiles included in the image, each of the one or more tiles being a tileoutside which a reference to is prohibited in inter-picture prediction.

The configuration described above allows the image transmissionapparatus to switch transmission of the entirety of the image andtransmission of a portion of the image in accordance with the type ofthe image. Thus, the image transmission apparatus enables the efficientprocessing of an encoded image.

For example, while the partial range of the image is selected, thetransmission unit may transmit the entirety of the image when the imagebelongs to the first type, the first type being a type in which displayof the entirety of the image is recommended, and the transmission unitmay transmit a portion of the image when the image belongs to the secondtype, the second type being a type in which display of the entirety ofthe image is not recommended.

The configuration described above allows the image transmissionapparatus to transmit the entirety of the image when display of theentirety of the image is recommend, and to transmit a portion of theimage when display of the entirety of the image is not recommend. Thus,the image transmission apparatus enables transmission of an appropriateimage.

For example, while the partial range of the image is selected, thetransmission unit may transmit the entirety of the image when the imagebelongs to the first type, the first type being a type indicating anadvertisement, and the transmission unit may transmit a portion of theimage when the image belongs to the second type, the second type being atype indicating a different from an advertisement.

The configuration described above allows the image transmissionapparatus to transmit the entirety of the image for an advertisement forwhich the entirety of the image is desirably displayed, and to transmita portion of the image for a main part of a broadcast program for whichthe entirety of the image may not necessarily be displayed.

For example, the image may be assigned a flag indicating whether theimage belongs to the first type or the second type, and the transmissionunit may transmit the entirety or portion of the image in accordancewith the flag.

The configuration described above allows the image transmissionapparatus to transmit the entirety or portion of the image in accordancewith the flag assigned to the image.

These general or specific aspects may be implemented using a system, anapparatus, a method, an integrated circuit, a computer program, or anon-temporary recording medium such as a computer-readable compact discread-only memory (CD-ROM), or may be implemented using any combinationof a system, an apparatus, a method, an integrated circuit, a computerprogram, and a recording medium.

Some embodiments will be specifically described hereinafter withreference to the drawings. Each of the following embodiments provides ageneral or specific example. The values, shapes, materials, constituentelements, the positions at which the constituent elements are located,the way in which the constituent elements are connected, steps, theorders of the steps, etc., in the following description areillustrative, and are not intended to limit the present disclosure. Inaddition, among the constituent elements in the following embodiments, aconstituent element not recited in any of the independent claimsindicating the most generic concept of the present disclosure isdescribed as optional.

Substantially the same elements are given the same numerals, and adescription thereof may be omitted.

First Embodiment

In a first embodiment, a description will be given of an imageprocessing system for decoding an image from a bit stream in which theimage is divided into a plurality of tiles and is encoded, a method fordecoding the image, and a method for generating the bit stream.

In particular, in the first embodiment, a bit stream includesinformation for enabling a parallel implementation of encoding anddecoding. For example, an image is divided into a plurality of areaseach referred to as a tile. A bit stream includes information that iscapable of specifying a constraint for motion prediction or motioncompensation such that a motion prediction process or a motioncompensation process includes a reference to only pixels in one or morecorresponding tiles in an image signal at the same time or image signalsat different times.

Explanation of Configuration

FIG. 3 illustrates an example configuration of an image processingsystem 100 according to the first embodiment. The image processingsystem 100 illustrated in FIG. 3 includes a decoding apparatus (an imagedecoding apparatus) 200 and a server (an image transmission apparatus)300. The decoding apparatus 200 includes a processing unit 210, astorage unit 220, a communication unit 230, a control unit 240, and adisplay unit 250. The server 300 includes a processing unit 310, astorage unit 320, a communication unit 330, and a control unit 340. Thestorage unit 320 stores image streams A, B, and C, which have beenencoded.

The storage unit 220 of the decoding apparatus 200 and the storage unit320 of the server 300 may be a memory or the like external thereto, ormay be an external memory connected via a wired or wireless network.Presently, the server 300 generally has a memory located in each countryof the world, which stores similar data, so that images are distributedfrom the nearest location in accordance with the position of thedecoding apparatus 200 by using the global positioning system (GPS) orthe like.

The other constituent elements may also be connected to one another in awired or wireless way so that the overall processing can be executed byhardware or software, and may not necessarily be physically integratedinto a single apparatus.

In addition, each component may be composed of one or more processors,one or more circuits, one or more memories, or any combination of them,or may be a general-purpose circuit or the like that executes softwareto perform the processing of each component.

FIG. 4 is a block diagram illustrating an example configuration of thedecoding apparatus 200 illustrated in FIG. 3. More particularly, FIG. 4illustrates an example configuration of the processing unit 210 of thedecoding apparatus 200. The decoding apparatus 200 illustrated in FIG. 4includes an entropy decoding unit 201, an inverse quantization unit 202,an inverse transform unit 203, a block memory 204, a frame memory 205,an intra prediction unit 206, and an inter prediction unit 207.

The entropy decoding unit 201, the inverse quantization unit 202, theinverse transform unit 203, the intra prediction unit 206, and the interprediction unit 207 are included in the processing unit 210. The blockmemory 204 and the frame memory 205 may be included in the storage unit220, or may be separate from the storage unit 220.

The processing unit 210 receives, as input, a bit stream, or an encodedimage signal, which is acquired by the communication unit 230 fromoutside.

The entropy decoding unit 201 performs entropy decoding on the bitstream. The inverse quantization unit 202 inverse-quantizes coefficientsobtained by entropy decoding. The inverse transform unit 203 performs aninverse frequency transform on the inverse-quantized coefficients. Aframe (or picture) which is composed of blocks obtained by inversefrequency transformation is output to the display unit 250.

The block memory 204 stores the blocks obtained by inverse frequencytransformation. The frame memory 205 stores the frame formed of theblocks obtained by inverse frequency transformation. The intraprediction unit 206 performs intra prediction with reference to theblocks stored in the block memory 204. The inter prediction unit 207performs inter prediction with reference to the frame stored in theframe memory 205.

FIG. 5 is a diagram illustrating an example of the syntax that specifiesa constraint for motion compensation. The bit stream includes the syntaxillustrated in FIG. 5.

FIG. 6 is a diagram illustrating an example of a bit stream includingthe syntax illustrated in FIG. 5. As illustrated in FIG. 6, the syntaxillustrated in FIG. 5 is written in a header called supplementaryenhancement information (SEI). SEI is a header containing auxiliaryinformation that is commonly used for subsequent encoded image data.Note that, as might be anticipated, the syntax illustrated in FIG. 5 maybe included in a header called a sequence parameter set (SPS) or apicture parameter set (PPS) or may be included in any other header.

The processing unit 210 writes necessary data to the block memory 204and the frame memory 205 or reads necessary data from the block memory204 and the frame memory 205. The entropy decoding unit 201, the inversequantization unit 202, the inverse transform unit 203, the intraprediction unit 206, and the inter prediction unit 207 in the processingunit 210 perform a decoding process as specified by H.26x to decode animage from the bit stream.

The communication unit 230 transmits a request to the outside totransmit a bit stream in accordance with instructions from theprocessing unit 210, the control unit 240, or the like, and receives thebit stream from the outside.

In response to the request from the decoding apparatus 200, the server300 reads the corresponding bit stream from the storage unit 320 andtransmits the image data processed by the processing unit 310 from thecommunication unit 330 to the decoding apparatus 200 under the controlof the control unit 340. For example, the server 300 transmits the imagestream A to the decoding apparatus 200 in response to a request for theimage stream A from the decoding apparatus 200.

Explanation of Operation

FIG. 7 is a diagram illustrating the processing flow of the operation ofthe image processing system 100 according to the first embodiment.

The decoding apparatus 200 transmits a request for an image which theuser wishes to view to the server 300 outside the decoding apparatus 200(S501). Presently, a processing apparatus that receives the request anda server that delivers the image to the decoding apparatus 200 inaccordance with instructions from the processing apparatus are generallydifferent. In addition, the processing apparatus and the server aregenerally run by different companies, and are also generally located indifferent countries. In the description provided herein, the apparatusthat receives the request, and the server or the like that delivers theimage are collectively termed the server 300, which is located outsidethe decoding apparatus 200, for simplicity.

The decoding apparatus 200 delivers additional information in additionto the request to the server 300, and may thus be able to receive a moreappropriate image. The additional information includes, for example,position information, user information, performance information, and soforth. The additional information is information that allows the server300, which stores several images having the same content but differentformats, to transmit a more appropriate image among the several imagesto the decoding apparatus 200.

The server 300 receives the request for the image (S502), and determineswhether or not additional information is present (S503).

If additional information is present (YES in S503), the server 300selects image data to be transmitted, in accordance with the additionalinformation (S504). If additional information is not present (NO inS503), the server 300 selects image data to be transmitted, inaccordance with predetermined conditions (S505). The image data includesa main image corresponding to the request from the decoding apparatus200, and sub-information including text or an image.

The server 300 transmits a bit stream including the selected image datato the decoding apparatus 200 (S506). The decoding apparatus 200receives the bit stream (S507). Then, the decoding apparatus 200 decodesan encoded image included in the bit stream, and displays the decodedimage (S508).

Detailed Explanation of Selection Process

FIG. 8 is a diagram illustrating an example of the image data selectionprocess (S503 to S505) illustrated in FIG. 7.

In determination operations (S601 to S603) illustrated in FIG. 8, it isdetermined whether or not the additional information includesperformance information, position information, and user information. Thedetermination operations may be performed in any order or may beperformed in parallel. Note that the determination operations are merelyan example. The image data selection process may involve any otherprocedure including one or more steps of selecting sub-information inaccordance with any information.

Not all of the determination operations for performance information,position information, and user information may be performed. Otherinformation may be used for the image data selection process, orsub-information may be selected in accordance with performanceinformation.

In FIG. 8, for example, if it is determined that the additionalinformation includes performance information (YES in S601), a standardwith which the decoding apparatus 200 is compatible for decodingoperations or the profile, level, or the like in the standard which issupported by the decoding apparatus 200 is determined in accordance withthe performance information. Examples of the standard include MPEG-2,MPEG-4 Visual, H.264/Advanced Video Coding (AVC), and H.265/HEVC. Theserver 300 selects one of a plurality of types of main images stored inthe storage unit 320 of the server 300 in accordance with thedetermination result (S604).

If the decoding apparatus 200 is compatible with a plurality ofstandards or is compatible with a plurality of profiles or the like, theserver 300 may select one main image complying with a higher quality andhigher efficiency standard or profile from among the plurality of mainimages further in accordance with states such as a transmission pathstate. Alternatively, the server 300 may use a distribution system(e.g., MPEG-Dynamic Adaptive Streaming over Hypertext Transfer Protocol(HTTP) (MPEG-DASH)) that delivers information linked to each main imagecandidate to distribute the information so that the decoding apparatus200 can select a main image in accordance with states such as atransmission path state.

If it is determined that the additional information does not includeperformance information (NO in S601), the server 300 selects apredetermined main image, for example, a main image complying with themost fundamental standard, profile, or the like (S605).

If it is determined that the additional information includes positioninformation (YES in S602), the server 300 selects sub-information inaccordance with a country or region identified by the positioninformation as the location of the decoding apparatus 200 (S606). Thesub-information is, for example, information indicating the weather ofthe region, radio programs or TV programs broadcast in the region,nearby grocery stores or restaurants in the region, or the like, whichis expressed in the language of the region.

If it is determined that the additional information does not includeposition information (NO in S602), the server 300 may not necessarilyselect sub-information based on a location, or may select predeterminedsub-information. For example, if it is difficult to acquire positioninformation and it is difficult to identify the optimum language, theserver 300 may select general news expressed in English assub-information.

If it is determined that the additional information includes userinformation (YES in S603), the server 300 selects sub-information inaccordance with the user information. For example, the server 300selects sub-information in accordance with the gender, age, or the likeof the user (S607). Alternatively, the server 300 may acquireinformation regarding the network of a friend with whom the user is wellacquainted via a social networking service (SNS), and may selectsub-information in accordance with the preference of the user, which isestimated from the acquired information.

For example, if the user is interested in automobiles, the server 300selects news on the latest products or an advertisement forautomobile-related goods regarding the exterior or interior ofautomobiles as sub-information.

If it is determined that the additional information does not includeuser information (NO in S603), the server 300 may not necessarily selectsub-information based on user information, or may select predeterminedsub-information. For example, the server 300 may select, assub-information, information related to a main image, such asinformation on goods or a movie related to the main image.

In the manner described above, the server 300 selects or generates oneor more of a plurality of sub-information candidates as sub-information.

Transmission Method of Main Image and Sub-Information

In transmission process (S506) illustrated in FIG. 7, an image includingthe main image and sub-information selected in the manner describedabove is constructed using a plurality of tiles, and a bit streamincluding the constructed image is transmitted to the decoding apparatus200. Each of the plurality of tiles is basically an MV-constrained tile.However, the plurality of tiles may include a tile that is not anMV-constrained tile.

FIG. 9 is a diagram illustrating an example of first allocation of amain image and sub-information to a plurality of tiles in an imageaccording to the first embodiment. As illustrated by way of example inFIG. 9, tiles are rectangular areas obtained by dividing one image (orpicture) by a vertical or horizontal straight line(s).

For example, as illustrated in FIG. 9, one image is divided into a firsttile and a second tile. Then, the processing unit 310 encodes theselected main image as the first tile and the selected sub-informationas the second tile in accordance with H.265/HEVC. In this case, theprocessing unit 310 encodes each of the first tile and the second tileas an MV-constrained tile. FIG. 9 illustrates an example in which, forexample, weather and news are selected as sub-information and areencoded so as to be arranged side by side.

FIG. 10 is a diagram illustrating an example of second allocation of amain image and sub-information to a plurality of tiles in an imageaccording to the first embodiment. FIG. 10 illustrates an exampleconfiguration of a main image and sub-information that is different fromthe example illustrated in FIG. 9. In the illustrated example, the firsttile and the fifth to eighth tiles correspond to sub-information, andthe second to fourth tiles correspond to a main image.

The first tile includes information related to the main image, such asbroadcast program information on the main image, and the fifth tileincludes an advertisement. In addition, the sub-information is dividedinto the sixth to eighth tiles so as to correspond to the main imagewhich is divided into the second to fourth tiles, respectively.

As illustrated in FIG. 10, in a case where each of the second to fourthtiles corresponding to the main image is associated with a particularperson or a microphone, object-based coding may be used as an audiocodec. If the sound at a specific position or the sound of a microphoneis reproducible during decoding, the decoding apparatus 200 mayreproduce audio corresponding to an image displayed in a tile. Inaddition, a link to audio adjusted so that the audio becomes the mainaudio may be inserted in the corresponding tile for the sub-information.

FIG. 11 is a diagram illustrating an example of third allocation of amain image and sub-information to a plurality of tiles in an imageaccording to the first embodiment. FIG. 11 illustrates an example inwhich a main image is divided into a first tile and a second tile andsub-information is divided into a third tile and a fourth tile. In theillustrated example, two kinds of sub-information, that is, weather andnews, are encoded as separate tiles.

FIG. 12 is a diagram illustrating an example of fourth allocation of amain image and sub-information to a plurality of tiles in an imageaccording to the first embodiment. FIG. 12 illustrates an example inwhich tiles that correspond to a portion of a main image are replacedwith sub-information and are encoded. Specifically, FIG. 12 illustratesan example in which a main image is divided into first to thirty-fifthtiles, among which the seventh tile and the twenty-ninth tothirty-second tiles are replaced with sub-information. In theillustrated example, an MV-constrained tile is used for the seventh tilein accordance with the period during which or the timing at which eachpiece of sub-information is displayed.

FIG. 13 is a diagram illustrating an example of the timing at which theallocation to a plurality of tiles in an image according to the firstembodiment is changed. Specifically, in FIG. 13, the manner of divisioninto tiles is switched at I pictures that are substantially at intervalsof predetermined duration.

For example, when weather information is to be displayed in the seventhtile for 5 seconds every 10 seconds, an MV-constrained tile is used forthe seventh tile of the picture for 5 seconds, and the weatherinformation is embedded in a portion of the main image. In FIG. 13, atI₀, the picture is divided into tiles so that the seventh tile iscreated. Then, the manner of division of the picture is maintained untilI₁. At I₁, the weather information is returned to the main image, andthe constraint and the manner of division are changed to those in theinitial state. This is merely an example, and the weather informationmay be returned to the main image while the manner of division of thepicture and the constraint are maintained.

If the manner of division of the picture is changed while the user isviewing the picture or if the allocation of the main image and thesub-information is changed, the server 300 notifies the decodingapparatus 200 of the change in the manner of division or the allocation.For example, if the server 300 uses a distribution system based onMPEG-DASH or the like, the server 300 modifies a Media PresentationDescription (MPD) file, and notifies the decoding apparatus 200 of themodification of the MPD file.

After that, at an I picture (in FIG. 13, I₃) that is substantially at anelapsed time of 10 seconds, the picture is again divided into tiles sothat the seventh tile is created. Then, the server 300 replaces aportion of the main image with sub-information, and encodes the image byusing the MV constraint.

FIG. 14 is a diagram illustrating an example of an operation of changingthe allocation to a plurality of tiles in an image according to thefirst embodiment. Specifically, FIG. 14 illustrates an example in whichsub-information displayed on the twenty-ninth to thirty-second tiles isreturned to the portion of the main image and the sub-informationdisplayed on the seventh tile is moved to the position of the first tileat the next timing in accordance with instructions from the user. Here,the next timing is, for example, the timing of an I picture appearingafter the instructions are given.

FIG. 15 is a diagram illustrating an example of the processing flow forchanging the allocation to a plurality of tiles in an image according tothe first embodiment. For example, as in the example illustrated in FIG.15, in accordance with instructions from the user, the decodingapparatus 200 transmits a request to the server 300 via thecommunication unit 230 to change the location where sub-information isto be displayed from the seventh tile to the first tile (S801 to S803).In response to the request, the server 300 transmits a bit stream inwhich the sequence of tiles has been changed so that the sub-informationis displayed in the first tile at the next appropriate timing.

Then, the decoding apparatus 200 receives the modified bit stream(S801). Then, the decoding apparatus 200 decodes an encoded imageincluded in the modified bit stream, and displays the decoded image(S804). When the user performs an operation of deleting the display ofthe sub-information (YES in S802), the decoding apparatus 200 transmitsa display change request corresponding to the deletion of thesub-information to the server 300 (S803). In this case, the server 300transmits a bit stream in which the sub-information has been returned tothe portion of the main image.

In FIG. 15, the decoding apparatus 200 sends the display change requestto the server 300. In a case where a distribution system capable ofswitching the stream to be received in accordance with the situation,such as MPEG-DASH, is used, the decoding apparatus 200 may select anappropriate image from among a plurality of options transmitted from theserver 300, without sending a request. The decoding apparatus 200 mayalso initially select a default image. When the server 300 hasinformation to be notified, such as emergency disaster information, theserver 300 may transmit the information so the information will bechosen.

It is to be anticipated that even in a case where such a distributionsystem as above is used, the decoding apparatus 200 may request theserver 300 to create an appropriate option if no appropriate optionexists.

In FIG. 15, the decoding apparatus 200 receives a new stream at the nexttiming after a display change request has been made. The decodingapparatus 200 may display any of the following in a tile for whichdisplay is to be changed, by way of example, until the reception anddecoding of a new stream are completed: (1) the currently displayeditem, and (2) a predetermined background image.

In a case where the currently displayed item is continuously displayed,stable display is obtained since the decoding apparatus 200 involves noadditional processing. On the other hand, the user may feel that thedisplay is less responsive because the display has not been changedalthough the user made a movement or deletion request.

Accordingly, if the period until the reception and decoding of a newstream are completed is short, the decoding apparatus 200 may displaythe currently displayed item. If the period until the reception anddecoding of a new stream are completed is equal to or longer than acertain period, it is desirable that the decoding apparatus 200 changethe operation so that the predetermined background image is displayed.

In a case where the predetermined background image is displayed, thedecoding apparatus 200 can visually notify the user that the request hasbeen accepted, by using the displayed background image. In the casewhere the predetermined background image is displayed, furthermore,processing is simple. Examples of the background image include a grayimage, a logo, a pattern image, an image stored in advance in a memory,and a tile portion of the main image that has been most recently (orlast) decoded and that is held in a memory. Furthermore, the backgroundimage is limited to a still image.

The decoding apparatus 200 may also perform operations other thanmovement and deletion, such as converting the sub-information from imageto text data, reducing the size of the area, and modifying thesub-information to other sub-information, in accordance with a useroperation. If the sub-information has a large amount of data andrequires heavy processing or in inconvenient cases such as whensub-information displayed as a moving image during rapid scene changesdistracts the viewer from concentrating on the main image, the decodingapparatus 200 may convert the sub-information to text data to achieve astate more suitable for viewing of the main image.

The user selects the source for movement by using a remote controllerand gives instructions to move the sub-information to the destination byusing an arrow key or the like, and the decoding apparatus 200 canaccept the instructions from the user. If the decoding apparatus 200 hasa touch panel, the user gives instructions to move the sub-informationfrom the source to the destination via touch gestures such as a swipe orgives instructions for deletion by swiping from inner to outer of thescreen, and the decoding apparatus 200 can accept the instructions fromthe user.

The decoding apparatus 200 can also accept instructions for movement ordeletion in a similar manner by using information on a movement or agesture detected by an acceleration sensor, a depth sensor, or the like.Here, an example in which the user gives instructions for movement ordeletion is illustrated by way of example. Instead, if sub-informationto be notified, such as emergency disaster information, is to bedisplayed on the decoding apparatus 200, the server 300 may select thesub-information by default.

As illustrated in FIGS. 9 to 12, in a case where a portion of a mainimage is replaced with sub-information and the sub-information isencoded as a tile in a picture, it is desirable that the sub-informationbe arranged in an area that does not interfere with the viewing of themain image, such as at any of the four corners or along any edge (left,right, top, or bottom) of the screen.

However, in a case where a main image provided for a fee is distributedfor a trial period, a tile for sub-information such as an advertisementmay be arranged in an area that interferes with the viewing of the mainimage. Accordingly, the decoding apparatus 200 can encourage the viewerto view a more appropriate main image. The decoding apparatus 200 thusachieves the provision of a content distribution service for which anappropriate copyright fee is collected from users.

FIG. 16 illustrates an example of an operation of scrambling some of aplurality of tiles in an image according to the first embodiment. Forexample, in FIG. 16, initially, all the tiles in the image correspond toa main image. Then, some of the tiles may be gradually modified toMV-constrained tiles at the timing when I pictures are displayed. In themodified tiles, sub-information, a gray image, a scrambled image, anindiscernible main image such as a coarse or blurred main image, or thelike may be displayed.

When the user is to view the main image, the user may select the desiredtile by using, for example, the user interface (UI) described above, andpay an appropriate viewing fee via an electronic payment system, so thatan indication that interferes with viewing on the tile may be canceled.The viewing fee may be determined in accordance with the size of thearea selected by the user as an area to view. A tile for displayingsub-information may be arranged in an unselected area, and anadvertisement or the like may be always displayed in the tile.

In the manner described above, the decoding apparatus 200 receives thetransmitted bit stream, reads the standard with which the bit streamcomplies from the header, and executes a decoding process correspondingto the standard to display a decoded image on the display unit 250.

On the other hand, the server 300 holds a plurality of bit streamscorresponding to a plurality of patterns having the same number of tilesbut having different pieces of content so as to meet a request from thedecoding apparatus 200 as described above. For example, in a case whereMPEG-DASH is employed, the server 300 may transmit an MPD file includingsegment information linked to the plurality of bit streams to thedecoding apparatus 200.

Advantages

In the first embodiment, as described above, an image is divided into aplurality of tiles, and a selected main image and selectedsub-information are encoded in separate tiles. Accordingly, an imageaccompanied by appropriate information is transmitted to the decodingapparatus 200 in the form of a single stream. Additionally, the server300 can transmit sub-information to be transmitted, and the decodingapparatus 200 can receive appropriate sub-information.

In the manner described above, the image processing system 100 accordingto the first embodiment can transmit and receive an image accompanied byappropriate information.

Supplemental Information

In the image encoding in the transmission process (S506) illustrated inFIG. 7, if both the main image and the sub-information are data encodedin advance in accordance with H.265/HEVC, the server 300 may combine themain image and the sub-information. Then, the server 300 rewrites theheader portion so that the total number of tiles, which is the sum ofthe number of tiles for the main image and the number of tiles for thesub-information, is included as the number of tiles in the image, sothat a bit stream including the main image and the sub-information isconstructed.

In contrast, for example, if the main image is data encoded inaccordance with H.264/AVC and the sub-information is text data, JointPhotographic Experts Group (JPEG) data, or the like, an image includingthe main image and the sub-information may be re-encoded so as to complywith H.265/HEVC in its entirety.

Even if both the main image and the sub-information are data encoded inadvance in accordance with H.265/HEVC, the server 300 may re-encode animage including the main image and the sub-information so as to avoid areference to outside the screen. For example, if a tile located in anedge portion of the main image includes a block that has a reference tooutside the screen (or a reference to outside the picture), merelyrewriting of only the header portion may cause a reference to a tile forthe sub-information. Thus, the server 300 may re-encode an imageincluding the main image and the sub-information so as to avoid areference to the outside the screen.

In some cases, the selected sub-information and a prepared area may notmatch. For example, the sub-information has a large amount of dataincluding a moving image and a tile that is small to such an extent thatit is difficult to visually observe a moving image is prepared for thesub-information. In these cases, the server 300 may convert thesub-information. Specifically, the server 300 may extract text data fromthe moving image in the sub-information, and the sub-information may bereplaced with the extracted text data before the main image and thesub-information are encoded.

FIG. 17 is a block diagram illustrating an example configuration of theprocessing unit 310 of the server 300 illustrated in FIG. 3. Theencoding process described above is performed by the processing unit 310having the configuration illustrated in FIG. 17. The processing unit 310illustrated in FIG. 17 includes a subtraction unit 301, a transform andquantization unit 302, an entropy encoding unit 303, an inversequantization and inverse transform unit 304, an adder unit 305, adeblocking filter unit 306, an intra prediction unit 307, a motioncompensation unit 308, a motion detection unit 309, and a memory 311.

The subtraction unit 301 subtracts a value indicated by a predictionsignal from a value indicated by an input signal to obtain a predictionerror, and outputs the prediction error. The transform and quantizationunit 302 performs a frequency transform and quantization on theprediction error to obtain quantization coefficients, and outputs thequantization coefficients. The entropy encoding unit 303 performsentropy encoding on the quantization coefficients to obtain an encodedsignal, and outputs the encoded signal. The inverse quantization andinverse transform unit 304 performs inverse quantization and an inversefrequency transform on the quantization coefficients. The adder unit 305adds together the prediction error obtained by inverse quantization andinverse frequency transformation and the value indicated by theprediction signal, and outputs a pixel value.

The deblocking filter unit 306 removes block noise in an image formed ofpixel values. The intra prediction unit 307 performs intra predictionwith reference to a pixel value in the picture to be encoded. The motioncompensation unit 308 performs inter prediction with reference to apixel value in a picture different from the picture to be encoded. Themotion detection unit 309 performs motion detection in accordance withthe input signal and a picture stored in the memory 311 to obtain motiondata, and outputs the motion data. The memory 311 stores a picturereconfigured using the sum of the prediction error and the valueindicated by the prediction signal or the like.

The memory 311 may be the storage unit 320 included in the server 300,or may be a memory specific to the processing unit 310.

If the decoding apparatus 200 does not support H.265/HEVC, the server300 may use a transmission method in which the main image and thesub-information are encoded using the corresponding encoding scheme andare packetized and delivered as separate streams. In this case, thedecoding apparatus 200 may display the main image and thesub-information so that the main image corresponds to the entire screenand the sub-information is superimposed on an area that is a portion ofthe main image. In this transmission method, furthermore, packetsincluding sub-information may be frequently transmitted to prevent theoccurrence of missing data.

In a case where H.265/HEVC is used, priority may be set in advance sothat sub-information of information regarded as having higher priority,such as typhoon information, may be more preferentially frequentlyembedded than, for example, advertisements. In addition, a request maybe received from the decoding apparatus 200 to exclude the display ofspecific information, and the selection operation may be controlled sothat the specific information is not selected in the subsequent image.

Second Embodiment

A second embodiment provides another example of a decoding method, adisplay method, and a management method in an image processing system100 having a configuration similar to that in the first embodiment. Alsoin the second embodiment, a bit stream includes information forspecifying a constraint for motion prediction or motion compensation.

In the second embodiment, when the user starts to view a main image, thedecoding apparatus 200 displays one or more modes on the display unit250 as one or more options in accordance with a user operation orautomatically. A mode corresponds to a manner of division of a screen(or a picture). That is, the decoding apparatus 200 displays how ascreen is divided into a plurality of tiles, for each of the one or moremodes, on the display unit 250.

FIGS. 18 to 20 illustrate an example of the display of individual modes.In FIGS. 18 to 20, an area displayed shaded is an area where a mainimage is mainly displayed. A user selects one mode from among one ormore modes. The decoding apparatus 200 receives a bit stream including amain image and sub-information from the server 300 in the manner ofdivision corresponding to the selected mode.

In the second embodiment, in the decoding apparatus 200, which is aterminal on which the user views the main image, a display area isdivided into a plurality of tiles in accordance with user preference.The user is able to view the main image using the desired tile. Further,for example, the decoding apparatus 200 superimposes the sub-informationon a portion of the main image so that any other application such asemail or a game appears in foreground. The decoding apparatus 200 maysimultaneously execute a plurality of applications.

In addition, selecting a mode at the start of viewing results insuppression of, during viewing, the occurrence of a request to changethe location where the sub-information is to be displayed. Thus, theserver 300 can transmit a stream originally held therein to the decodingapparatus 200 in accordance with one or more modes without changing theencoding format.

The decoding apparatus 200 displays individual modes so that the usermay easily identify which mode is suitable for viewing. For example, thedecoding apparatus 200 may show, for each mode, tile boundaries asbroken lines or the like, the boundaries between the area for the mainimage and the area for the sub-information as bold lines, or the area ofthe main image and the area of the sub-information using differentcolors.

In the foregoing, a user selects a mode. Alternatively, the server 300may determine an initial mode, which corresponds to division into tilesand allocation to tiles, in accordance with the content of the mainimage or environments such as the degree of distribution networktraffic. Then, the decoding apparatus 200 may display the main image andthe sub-information in the initial mode.

Accordingly, the server 300 allows the user to view the main image andthe sub-information with more appropriate division and allocation inaccordance with the content of the main image and so forth. In addition,unauthorized deletion or alteration of sub-information to be viewed,such as emergency disaster information, may be suppressed. Further, thedecoding apparatus 200 may display the latest MPD file, and may acceptthe input of a reselected mode from the user using a modification in theMPD file as a trigger.

Third Embodiment

A third embodiment provides another example of a decoding method, adisplay method, and a management method in an image processing system100 having a configuration similar to that in the first embodiment. Alsoin the third embodiment, a bit stream includes information forspecifying a constraint for motion prediction or motion compensation.

As illustrated in the first and second embodiments, in a case where thedisplay method of a main image and sub-information is changed inaccordance with a user operation, the decoding apparatus 200 or theserver 300 may hold information on the user operation or the viewingtype. Further, the decoding apparatus 200 may display an image byreceiving and decoding a bit stream for displaying more appropriatesub-information in accordance with the held information.

For example, a user may perform an operation of enlarging anadvertisement for automobiles while performing an operation of deletingor reducing other advertisements, or may select a tile in which anadvertisement for automobiles is displayed and access a uniform resourcelocator (URL) defined in an MPD file or the like. In this case, thedecoding apparatus 200 may request the server 300 to generate a bitstream that shows a large number of advertisements for automobiles.Alternatively, the server 300 may initially select the above-describedbit stream.

Fourth Embodiment

A fourth embodiment provides another example of an image processingsystem for decoding an image from a bit stream in which the image isdivided into a plurality of tiles and is encoded, a method for decodingthe image, and a method for generating the bit stream.

In particular, in the fourth embodiment, the bit stream includesinformation for enabling a parallel implementation of encoding anddecoding. For example, an image is divided into a plurality of areaseach referred to as a tile. The bit stream includes information that iscapable of specifying a constraint for motion prediction or motioncompensation such that a motion prediction process or a motioncompensation process includes a reference to only pixels in one or morecorresponding tiles in an image signal at the same time or image signalsat different times.

Explanation of Configuration

FIG. 21 illustrates an example configuration of a broadcast system 110according to the fourth embodiment. A television (TV) 400, a set-top box(STB) 460, a reader/recorder 470, a reproducing apparatus 600, a mobilephone (or smartphone) 700, and a car navigation system 800 eachcorrespond to the decoding apparatus 200 in the first embodiment. Amonitor 450 and a monitor 650 each correspond to the display unit 250 inthe first embodiment. A recording medium 430, a recording medium 431,and a recording medium 432 each correspond to the storage unit 220 inthe first embodiment.

A broadcast station 500 corresponds to the server 300 in the firstembodiment. One of the TV 400, the STB 460, the reader/recorder 470, thereproducing apparatus 600, the mobile phone (or smartphone) 700, and thecar navigation system 800 may be included in the broadcast system 110 asan apparatus that receives a broadcast. That is, not all of theabove-described apparatuses and devices may be included in the broadcastsystem 110 as apparatuses that receive a broadcast.

The broadcast station 500 transmits multiplexed data, which is obtainedby multiplexing encoded music data or the like onto encoded video data,to a satellite 501 by radio wave. Upon receipt of the radio wave fromthe broadcast station 500, the satellite 501 sends a broadcasting radiowave, and a home antenna 503 capable of receiving satellite broadcastsreceives the broadcasting radio wave. The multiplexed data may betransmitted via a cable 502. An apparatus such as the TV (receiver) 400or the STB 460 decodes the received multiplexed data, displays an imageon the monitor 450, and outputs audio from a speaker.

An antenna 802 of a vehicle 801 may receive the radio wave. In thiscase, the car navigation system 800 displays an image in a mannersimilar to that of the TV 400.

In addition, the reader/recorder 470, which includes the processingunits 210 and 310 (i.e., a decoding processing unit and an encodingprocessing unit) illustrated in FIG. 4 and FIG. 17, records andreproduces the received multiplexed data onto and from the recordingmedium 431 or 432, examples of which include a hard disk drive (HDD), adigital versatile disc (DVD), a Blu-ray Disc (BD), and a Secure Digital(SD) card. The decoded video and audio are output via the monitor 450 orvia a speaker. In addition, another apparatus or another system canreproduce video and audio via the recording medium 431 or 432 having themultiplexed data recorded thereon.

In addition, the STB 460 connected to the cable 502 for cable TV orconnected to the antenna 503 for satellite/terrestrial broadcasts mayhave mounted therein the processing unit 210 illustrated in FIG. 4, andvideo may be displayed on the monitor 450 of the TV 400 via the STB 460.In addition, the TV 400 may have incorporated therein the processingunit 210 illustrated in FIG. 4.

FIG. 22 is a block diagram illustrating an example configuration of theTV 400. A tuner 401, a modulation/demodulation unit 402, and amultiplexing/demultiplexing unit 403 receive a multiplexed stream,demodulate the multiplexed stream, and separate the stream into audiodata and video data. A signal processing unit 420 including an audiosignal processing unit 404 and a video signal processing unit 405performs a decoding process on the audio data and the video data, andthe audio data and the video data are output from a speaker 480 and adisplay unit 450 in an output unit 409.

The TV 400 includes a power supply circuit unit 411 for supplying powerto each unit of the TV 400. The TV 400 further includes an LSI device900, and some constituent elements are included in the LSI device 900.

Further, the TV 400 is controlled by an external remote controller 490or the like, and the TV 400 further includes an interface unit 417 forcontrolling an external peripheral device. The interface unit 417includes an operation input unit 412, a bridge 413, a slot unit 414, adriver 415, and a modem 416. A plurality of other constituent elementsalso cooperate with each other under control of a control unit 410 toperform receiving, decoding, and display operations.

The control unit 410, the signal processing unit 420, and the displayunit (or monitor) 450 correspond to the control unit 240, the processingunit 210, and the display unit 250 in the first embodiment,respectively. The recording medium 431, the recording medium 432, abuffer 433, a buffer 434, a buffer 435, and a buffer 436 each correspondto the storage unit 220 in the first embodiment.

Explanation of Operation

FIG. 23 illustrates an example of the processing flow of the broadcastsystem 110 according to the fourth embodiment. First, the broadcaststation 500 broadcasts a plurality of image streams corresponding to aplurality of pieces of content without receiving a request from the TV400 (S1301). Each of the plurality of image streams is a stream inwhich, for example, one main image corresponding to one channel and thesub-information illustrated in the first embodiment are encoded using aplurality of tiles into which a screen (or a picture) is divided.

In the first embodiment, sub-information is selected from among aplurality of candidates in accordance with additional informationobtained from the decoding apparatus 200. In the fourth embodiment,sub-information is selected in advance from among a plurality ofcandidates.

Here, the sub-information is encoded as a tile. The sub-informationmainly includes high-priority information as illustrated in the firstembodiment. The sub-information may include information including aheadline in which low-priority information is summarized. The detailedcontent of the sub-information may be associated with a tile and may bedelivered as an additional separate stream.

The TV 400 checks an image stream corresponding to the content to beviewed (S1302). Then, the TV 400 determines whether or not there are aplurality of image streams corresponding to the content to be viewed(S1303). For example, the TV 400 determines whether or not there are aplurality of candidates for the main image which correspond to aplurality of different coding standards or a plurality of differentcoding levels. The TV 400 further determines whether or not there are aplurality of candidates for the sub-information encoded as a tile.

If there are a plurality of candidates for the main image (YES inS1303), the TV 400 selects a candidate capable of being received anddecoded with high quality and high efficiency as a main image. If thereare a plurality of candidates for the sub-information, the TV 400selects sub-information from among the plurality of candidates inaccordance with the position information, user information, or the likeheld therein. For example, the selection of sub-information is similarto that in the example illustrated in the first embodiment. Accordingly,the TV 400 selects an image stream in accordance with unique information(S1304).

If it is determined that there are not a plurality of image streams (NOin S1303), the TV 400 selects a receivable image stream (S1305). Then,the TV 400 decodes the image stream selected in the selection process(S1304, S1305), and displays the decoded image stream on the monitor 450(S1306).

Advantages

In the manner described above, the broadcast station 500 divides ascreen (or a picture) into a plurality of tiles, and encodes a selectedmain image and selected sub-information as different tiles.

Accordingly, the TV 400 can receive an image accompanied by appropriateinformation in the form of a single stream. In addition, the broadcaststation 500 can transmit sub-information so that the sub-informationwill not be altered by an apparatus or system that receives or relaysthe sub-information into an unintended form. Thus, the TV 400 canreceive appropriate sub-information.

In the manner described above, the TV 400 in the fourth embodiment canreceive an image accompanied by appropriate information.

Supplemental Information

The selection method, encoding method, decoding method, transmissionmethod, and display method illustrated in the first to third embodimentsare also applicable to the fourth embodiment.

For example, the quality of the main image may be changed, the viewingchannel may be switched, a commercial break (hereinafter sometimesreferred to as a “CM”) may be inserted, or a user may give instructionsto directly change sub-information. If the MPEG2 system is used, anotification of a change in a tile constituting content may be sentusing program information, event information, or the like. The TV 400may accept reselection of a tile (for sub-information or the like) byusing the change of the tile as a trigger.

A piece of content is broadcast in association with a channel. However,in an image having a large size, such as an 8k4k image, a piece ofcontent may be broadcast on a plurality of channels. In addition, aportion of an image may be transmitted and received via broadcasting,and a portion of an image may be transmitted and received viacommunication. That is, an image may be transmitted and received using aplurality of transmission paths.

In some cases, one candidate for the main image (or one main image) maybe present on one channel. In other cases, a plurality of candidates forthe main image which have different coding standards or levels may bepresent on one channel.

Fifth Embodiment

A fifth embodiment provides another example of an image processingsystem for decoding an image from a bit stream in which the image isdivided into a plurality of tiles and is encoded, a method for decodingthe image, and a method for generating the bit stream.

In particular, also in the fifth embodiment, a bit stream includesinformation for enabling a parallel implementation of encoding anddecoding. For example, an image is divided into a plurality of areaseach referred to as a tile. The bit stream includes information that iscapable of specifying a constraint for motion prediction or motioncompensation such that a motion prediction process or a motioncompensation process includes a reference to only pixels in one or morecorresponding tiles in an image signal at the same time or image signalsat different times.

Overview

The HEVC scheme and the like enable transmission and reception of imagedata with large format and high definition, such as 8k4k. However, amobile terminal that receives the image data may not necessarily besuitable to display a large image. In addition, a user may be interestedin only a portion of a large image, and may wish to view the portion ofthe image in detail. Furthermore, such an area may change with time.

In addition, some image data may be less affected by the viewing of onlya portion of the image. Other image data may be data in which an imagewill be difficult to understand if the entirety of the image is notviewed.

In the fifth embodiment, a partial range of an image is selected, andappropriate decoding and appropriate display are performed. Morespecifically, in a case where content including both image data that issuitable for viewing a clipped portion of an image, such as image datafor a soccer game, and image data suitable for viewing an entire image,such as image data for a commercial advertisement, is received andviewed, comfortable viewing is achieved.

Explanation of Configuration

A configuration according to the fifth embodiment is similar to that ofthe first embodiment (FIG. 3 and FIG. 4), and is not described here.

Explanation of Operation

FIG. 24 and FIG. 25 are diagrams illustrating the processing flow of theoperation of the image processing system 100 according to the fifthembodiment.

The decoding apparatus 200 transmits a request for an image to be viewedto the server 300 (S1401). More specifically, the decoding apparatus 200requests the server 300 to transmit an image stream (or an imagesequence) including a plurality of images to be viewed. The server 300receives the request for the image (S1402).

Further, the server 300 determines whether or not a selection range (orselection position) that is the range of an area to be reproduced in theimage has been saved (S1403). If the selection range has not been saved(NO in S1403), the server 300 requests the decoding apparatus 200 totransmit the selection range (S1403). Upon receipt of the request forthe selection range (YES in S1405), the decoding apparatus 200 transmitsthe selection range to the server 300 (S1406). The server 300 receivesthe selection range, and saves the received selection range (S1407).

Then, the server 300 determines whether or not an image to betransmitted within the image stream is an entire-display-intended image(S1408). The entire-display-intended image is an image suitable fordisplay in its entirety. If the image is an entire-display-intendedimage (YES in S1408), the server 300 selects the entirety of the imageregardless of the selection range specified by the decoding apparatus200 (S1409).

If the image to be transmitted is not an entire-display-intended image(NO in S1408), the server 300 acquires the selection range saved in thesaving process (S1407) or a selection range which has been previouslysaved (S1410). Then, the server 300 selects a portion of the image inaccordance with the acquired selection range (S1411).

Then, the server 300 transmits a bit stream including the selectedentirety or portion of the image to the decoding apparatus 200 (S1412).The decoding apparatus 200 receives the bit stream transmitted from theserver 300 (S1413).

Similarly to the first embodiment, the transmitted and received bitstream is generated by the encoding of the image using an MV-constrainedtile. As illustrated in FIG. 2, the block to be processed in theMV-constrained tile has a reference to a reference area in the range ofthe corresponding tile for a temporally different image (that is, foranother temporally different picture).

For this reason, for example, if the selection range is the second tileillustrated in FIG. 2, an entire bit stream corresponding to theentirety of the image may not necessarily be transmitted and received.In this case, only a portion of the bit stream corresponding to thesecond tile may be transmitted and received. The decoding apparatus 200can decode and display an image in the second tile in accordance withthe portion of the bit stream corresponding to the second tile.

Then, the decoding apparatus 200 decodes the image from the bit stream.Then, the decoding apparatus 200 resizes (i.e., enlarges or reduces) thedecoded image so that the size of the decoded image can fit the displaysize of the decoding apparatus 200, and displays the resulting image(S1414).

If the decoding apparatus 200 receives instructions from the user tostop reproduction (YES in S1415), the decoding apparatus 200 transmits areproduction stop request to the server 300 (S1416). Otherwise (NO inS1415), the decoding apparatus 200 repeatedly performs the receiving,decoding, and displaying operations (S1413 and S1414) until the displayof all the images in the image stream is completed (S1418).

Further, the server 300 repeatedly performs the selection andtransmission operations (S1408 to S1412) until the server 300 receives areproduction stop request from the decoding apparatus 200 or until thetransmission of all the images in the image stream is completed (S1417).

Advantages

With the operations described above, for an entire-display-intendedimage, reproduction of the image is automatically switched toreproduction of the entirety of the image regardless of the selectionrange. Thus, user convenience may be improved.

More specifically, when selecting the entirety or a portion of an image,the server 300 determines whether or not the image is anentire-display-intended image. If the image is anentire-display-intended image, the server 300 selects the entirety ofthe image. If the image is not an entire-display-intended image, theserver 300 selects a portion of the image in accordance with theselection range specified and saved by the user or in accordance with aselection range specified during previous reproduction. Accordingly,selection is automatically switched. A more specific example will now bedescribed with FIGS. 26 to 28.

FIG. 26 is a diagram illustrating an example of the timing at whichentire display and partial display are switched. Specifically, FIG. 26illustrates an example of an image stream selected by a user. In theillustrated example, the image stream has three commercial breaks (oradvertisement portions) between four scenes (or main part portions). Theimages included in the first to fourth main part portions are notentire-display-intended images, and the images included in the first tothird advertisement portions are entire-display-intended images.

FIG. 27 is a diagram illustrating an example of partial display.Specifically, FIG. 27 illustrates an example of a portion of the imagein the first main part portion of the image stream specified by theuser, and illustrates an example in which the range of the upper leftsix tiles in the entire image illustrated in FIG. 12 is selected.

FIG. 28 is a diagram illustrating an example of entire display.Specifically, FIG. 28 illustrates an example of the image of the firstadvertisement portion. The image of the first advertisement portion mayor may not be divided into a plurality of tiles. Hereinafter, theoperation of the image processing system 100 will be described withreference to FIG. 26 to FIG. 28.

First, the decoding apparatus 200 selects the image stream illustratedin FIG. 26 in accordance with a user operation, and requests the server300 to transmit the selected image stream. The decoding apparatus 200accepts a selection range from the user so that the portion illustratedin FIG. 27 is reproduced, and transmits the selection range to theserver 300.

The server 300 transmits a bit stream indicating the portion of theimage illustrated in FIG. 27 for the first main part portion illustratedin FIG. 26. The server 300 further transmits a bit stream indicating theentirety of the image illustrated in FIG. 28 at the time when the imageto be processed is switched to the image of the first advertisementportion. The decoding apparatus 200 displays an image obtained bydecoding a bit stream. In some cases, the size of the image illustratedin FIG. 27 or FIG. 28 may not necessarily match the size of the displayunit 250 of the decoding apparatus 200. In those cases, the decodingapparatus 200 enlarges or reduces the image before displaying the image.

The content of a commercial break (or advertisement portion) isgenerally understood when the entirety of the image of the commercialbreak is viewed. For this reason, the entirety of an image for acommercial break is displayed. For the other scenes (or the main partportions), in contrast, only a portion of the image is displayed inaccordance with the display size of the decoding apparatus 200 or inaccordance with a request from a user. Accordingly, reproduction of aportion of the image and reproduction of the entirety of the image areautomatically switched without the user giving instructions. Thus, userconvenience may be improved.

If the image is not an entire-display-intended image, only a bit streamindicating a portion of the image is transmitted and received. Thus, ablock in an image that is not an entire-display-intended image may notnecessarily be decoded. In addition, the selection range of a portion ofthe image changes in accordance with instructions from the decodingapparatus 200 to the server 300 or in accordance with a selection rangethat has been previously saved. For this reason, a block in an imagethat is an entire-display-intended image is encoded without anyreference to a block in an image that is not an entire-display-intendedimage.

For example, if the image illustrated in FIG. 12 is not anentire-display-intended image, a portion (or range) that undergoesdecoding changes in accordance with instructions from the decodingapparatus 200 or in accordance with the saved selection range. Thus, insome cases, only the first tile may be decoded, and, in other cases,only the second tile may be decoded, depending on the situation. Afterthat, when an entire-display-intended image as illustrated in FIG. 28 isdecoded, a decoded portion of the image illustrated in FIG. 12 differsdepending on the situation. Thus, a reference from a block in theentire-display-intended image illustrated in FIG. 28 to the imageillustrated in FIG. 12 is prohibited.

In addition, a selection range in an image initially specified by theuser may be saved in the server 300. The saved selection range may becalled and used at the time of a transition from an advertisementportion to a main part portion. This may eliminate the need for the userto specify a selection range again each time reproduction of anadvertisement portion is completed.

Specifically, in the example illustrated in FIG. 26, the server 300reads a selection range initially specified by the user at the time of atransition of the image from the first advertisement portion to thesecond main part portion, and transmits a bit stream for the second mainpart portion in accordance with the selection range. Similarly, theserver 300 transmits the entirety of the image for the secondadvertisement portion, transmits a portion of the image for the thirdmain part portion, transmits the entirety of the image for the thirdadvertisement portion, and transmits a portion of the image for thefourth main part portion.

That is, the server 300 acquires a selection range initially specifiedby the user. After that, the server 300 automatically switches betweenreproduction of a portion of the image and reproduction of the entiretyof the image without acquiring a new selection range. Thus, userconvenience may be improved. In addition, the amount of processing forthe transmission and reception of information on a selection range fromthe decoding apparatus 200 to the server 300 may be reduced.

In a case where reproduction of the image is temporarily stopped inaccordance with a user operation and the reproduction of the image isresumed after a certain period of time, or in a case where reproductionof the same image is started using another decoding apparatus,re-specification of a selection range is omitted. The above-describedoperations will be specifically described with reference to FIG. 24 etc.

Upon receipt of a request for an image, the server 300 determineswhether or not a selection range has been saved (S1403). If a selectionrange has been saved (YES in S1403), the server 300 does not request thedecoding apparatus 200 to transmit a selection range. That is, ifanother decoding apparatus has specified a selection range and hasreproduced the same image, the selection range has been saved. Thus, theserver 300 does not make a request for any selection range.

For example, for the first main part portion in the image streamillustrated in FIG. 26, reproduction of a portion of the image asillustrated in FIG. 27 is started using a mobile phone in response to auser operation. After that, the reproduction is temporarily stopped inthe middle of the first main part portion in response to a useroperation.

Then, when the reproduction is resumed using a personal computer (PC) inresponse to a user operation, the server 300 calls the saved selectionrange, and transmits a bit stream indicating a portion of the image tothe PC in accordance with the saved selection range. Then, reproductionof a portion of the image as illustrated in FIG. 27 is started using thePC without receipt of instructions for any selection range from theuser.

This may prevent a selection range from being repeatedly specified.Thus, user convenience may be improved. In addition, the amount ofprocessing for the transmission and reception of information on aselection range from the decoding apparatus 200 to the server 300 may bereduced.

Supplemental Information

The server 300 determines whether or not the image is anentire-display-intended image (S1408 in FIG. 25). The determination ofwhether or not the image is an entire-display-intended image may bebased on an explicit specification in SEI or a PPS in the bit stream.

For example, a flag indicating for each image whether or not the imageis an entire-display-intended image may be included in a bit stream. Theflag may be a flag indicating whether or not to recommend display of theentirety of the image, or may be a flag indicating whether or not toprohibit display of a portion of the image. Specifically, if the flagindicates that display of the entirety of the image is recommended ordisplay of a portion of the image is prohibited, the server 300determines that the image is an entire-display-intended image.Otherwise, the server 300 determines that the image is not anentire-display-intended image.

In addition, for example, the server 300 may extract a feature of theimage and determine whether or not the image is anentire-display-intended image in accordance with the extracted featurein order to reduce the amount of coding or the amount of processing forexplicit specification of whether or not the image is anentire-display-intended image.

Furthermore, for a similar purpose, in cases such as when an image isnot divided into a plurality of tiles, when the number of tiles issmall, or when the manner of division of the current scene is differentfrom that for the preceding scene, the server 300 may determine that theimage is an entire-display-intended image. An image that is anentire-display-intended image and an image that is not anentire-display-intended image have largely different content. For thisreason, the manner of division into tiles is generally changed at thetime of a transmission between those images. Accordingly, thedetermination of whether or not the image is an entire-display-intendedimage may be based on a change in the manner of division.

In the example illustrated in FIG. 26, it is determined that the imagesof the commercial breaks are entire-display-intended images. Thedetermination of whether or not an image is an entire-display-intendedimage is not limited to that described above. For example, images ofemergency news, images of dramas, low-resolution images, and the likemay be determined to be entire-display-intended images. For example,images of sports, images of concerts, and the like may be determined notto be entire-display-intended images.

Alternatively, whether or not an image is an entire-display-intendedimage may be set as desired by the author or provider of the image.Setting an image whose entirety is desirably presented to a user as anentire-display-intended image allows the image to be displayed inaccordance with the intention of the author or provider of the image.

In the foregoing, a selection range is specified when reproduction isstarted. The selection range may be changed during reproduction inaccordance with a user operation or in accordance with the displaydevice. In this case, a newly specified selection range is saved again.

In the foregoing, furthermore, if no selection range is saved, theserver 300 requests the decoding apparatus 200 to transmit a selectionrange (S1403, S1404). Alternatively, if no selection range is saved,first, reproduction may be started by using an entire range of the imageas a selection range. After that, the selection range may be changed inaccordance with instructions for a selection range from the user.Alternatively, reproduction may be started by using a default selectionrange.

Accordingly, the amount of processing required when reproduction isstarted may be reduced. In addition, if display of the entirety of theimage or the default selection range meets the user's demand, thespecification of a new selection range can be omitted.

In addition, the server 300 may save an image selection range for eachuser. For example, the server 300 saves a selection range for each userID. If the same user is to reproduce the same image, the server 300 maycall the selection range corresponding to the user. Accordingly, even ifa plurality of users access the server 300 and reproduce an image,reproduction of the image is started with a selection range previouslyspecified by each individual user. Thus, user convenience may beimproved.

The previously specified selection range may be a range determined foran image using absolute coordinates, or may be a range determined on thebasis of any other criterion. For example, in a case where one of themembers of a pop group is selected in a video clip of live performanceof the pop group, an area where the selected member is displayed foreach image may be used as a selection range.

In addition, a selection range may be shared by users. Reproduction maybe started using a selection range specified by another user. That is, auser may use a selection range specified by another user withoutspecifying a selection range. This may allow a user to use anappropriate selection range, which is selected by another user, withoutany time-consuming specification.

For example, the image processing system 100 may identify interests,career, or the like of the user in accordance with information such asan SNS account, and recommend a selection range specified by anotheruser who has similar interests, career, or the like.

In the foregoing, furthermore, a selection range is saved in the server300. The selection range may be saved in an apparatus different from theserver 300, or may be saved in the decoding apparatus 200.

For example, in a case where a selection range is saved in the decodingapparatus 200, the decoding apparatus 200 may call the saved selectionrange when reproduction is started in accordance with a user operation,and transmit the selection range to the server 300. In a case where aselection range is saved in a different apparatus, the decodingapparatus 200 may read the selection range from the different apparatus,and transmit the selection range to the server 300. Alternatively, theserver 300 may read a selection range from a different apparatus inresponse to a request.

Here, reproduction of an image is illustrated. If a portion of an imageis to be displayed, the decoding apparatus 200 may decode only an audioportion related to the portion depending on the content of the image.

For example, in a case where, in an image on which a large number ofpersons appear, each individual person has a separate microphone and aselection range is a range where a specific person appears, the decodingapparatus 200 may decode only a signal from the microphone correspondingto the specific person. In this case, at the time of a transition of theimage to an entire-display-intended image, switching for the audio to bedecoded is also simultaneously performed so that the audio associatedwith the entirety of the image is decoded.

Conclusion

The characteristic configuration and characteristic operation of theimage processing system illustrated in the fifth embodiment will begiven hereinafter.

FIG. 29 is a block diagram illustrating, as an example, a characteristicconfiguration of an image processing system according to the fifthembodiment. An image processing system 1000 illustrated in FIG. 29includes an image decoding apparatus 1100 and an image transmissionapparatus 1200. The image decoding apparatus 1100 and the imagetransmission apparatus 1200 correspond to the decoding apparatus 200 andthe server 300 in FIG. 3, respectively.

The image decoding apparatus 1100 includes a selection unit 1101, atransmission unit 1102, a request unit 1103, a receiving unit 1104, adecoding unit 1105, a display control unit 1106, a display device 1107,and a saving unit 1110. In particular, the selection unit 1101, thedisplay control unit 1106, the display device 1107, the saving unit1110, and so forth may not necessarily be included in the image decodingapparatus 1100. For example, the selection unit 1101, the displaycontrol unit 1106, the display device 1107, the saving unit 1110, and soforth may be included in a different apparatus, or may not necessarilybe included in the image processing system 1000.

The selection unit 1101, the request unit 1103, and the decoding unit1105 mainly correspond to the processing unit 210 illustrated in FIG. 3.The transmission unit 1102 and the receiving unit 1104 mainly correspondto the communication unit 230 illustrated in FIG. 3. The display controlunit 1106 mainly corresponds to the control unit 240 illustrated in FIG.3. The display device 1107 mainly corresponds to the display unit 250illustrated in FIG. 3. The saving unit 1110 mainly corresponds to thestorage unit 220 illustrated in FIG. 3.

The image transmission apparatus 1200 includes a receiving unit 1201, arequest acceptance unit 1202, a transmission unit 1203, and a savingunit 1210. In particular, the saving unit 1210 may not necessarily beincluded in the image transmission apparatus 1200. For example, thesaving unit 1210 may be included in a different apparatus, or may notnecessarily be included in the image processing system 1000.

The receiving unit 1201 and the transmission unit 1203 mainly correspondto the communication unit 330 illustrated in FIG. 3. The requestacceptance unit 1202 mainly corresponds to the processing unit 310illustrated in FIG. 3. The saving unit 1210 mainly corresponds to thestorage unit 320 illustrated in FIG. 3.

FIG. 30 is a diagram illustrating, as an example, the processing flow ofthe characteristic operation of the image processing system 1000illustrated in FIG. 29. The image processing system 1000 processes anencoded image in accordance with the operation illustrated in FIG. 30.

In addition, the image decoding apparatus 1100 included in the imageprocessing system 1000 receives an encoded image from the imagetransmission apparatus 1200, and decodes the image received from theimage transmission apparatus 1200 in accordance with the operationillustrated in FIG. 30. In addition, the image transmission apparatus1200 included in the image processing system 1000 transmits an encodedimage to the image decoding apparatus 1100 in accordance with theoperation illustrated in FIG. 30.

Specifically, first, in the image decoding apparatus 1100, the selectionunit 1101 selects a partial range of the image within an entire range ofthe image in accordance with a user operation (S1101). For example, theselection unit 1101 accepts user input, and selects a partial range ofthe image in accordance with the accepted input. This operation may beomitted if a partial range of the image has already been selected.

The selection unit 1101 may inform each component of the image decodingapparatus 1100 of the partial range of the image via the saving unit1110. The saving unit 1110 saves range information indicating thepartial range of the image. The saving unit 1110 may save the rangeinformation in an internal storage unit of the image decoding apparatus1100, or may save the range information in a storage unit outside theimage decoding apparatus 1100. For example, the saving unit 1110 maysave the range information in the image transmission apparatus 1200 viathe transmission unit 1102.

Then, the request unit 1103 requests the image transmission apparatus1200 to transmit an image (S1102). For example, the request unit 1103transmits a signal to request transmission of an image to the imagetransmission apparatus 1200 via the transmission unit 1102.

Then, in the image transmission apparatus 1200, the request acceptanceunit 1202 accepts the request from the image decoding apparatus 1100 totransmit an image (S1201). For example, the request acceptance unit 1202receives a signal to request transmission of an image from the imagedecoding apparatus 1100 via the receiving unit 1201.

Then, in the image decoding apparatus 1100, the transmission unit 1102transmits range information indicating the partial range of the image tothe image transmission apparatus 1200 (S1103). More specifically, therange indicated by the range information is a range selected by theselection unit 1101 in accordance with a user operation.

Then, in the image transmission apparatus 1200, the receiving unit 1201receives the range information from the image decoding apparatus 1100(S1202). The saving unit 1210 may save the range information received bythe receiving unit 1201. The saving unit 1210 may also save the rangeinformation in an internal storage unit of the image transmissionapparatus 1200, or may save the range information in a storage unitoutside the image transmission apparatus 1200.

Then, in the image transmission apparatus 1200, the transmission unit1203 transmits the entirety or portion of the image to the imagedecoding apparatus 1100 while the partial range of the image is selected(S1203).

Specifically, if the image belongs to a first type, the transmissionunit 1203 transmits the entirety of the image to the image decodingapparatus 1100. If the image belongs to a second type, the transmissionunit 1203 transmits the portion of the image to the image decodingapparatus 1100. More specifically, if the image belongs to the secondtype, the transmission unit 1203 transmits only one or more tilescorresponding to the partial range of the image among the plurality oftiles included in the image, each of the one or more tiles being a tileoutside which a reference to is prohibited in inter-picture prediction.

For example, the first type is a type in which display of the entiretyof the image is recommend, and the second type is a type in whichdisplay of the entirety of the image is not recommend. For example, thefirst type is an advertisement, and the second type is not anadvertisement. For example, a flag indicating whether the image belongsto the first type or the second type may be attached to the image. Thetransmission unit 1203 may transmit the entirety or portion of the imagein accordance with the flag.

Each of one or more tiles outside which a reference to is prohibited ininter-picture prediction corresponds to an MV-constrained tile. A rangecorresponding to each tile is a range defined for each tile, and isbasically the range of each tile.

Then, in the image decoding apparatus 1100, the receiving unit 1104receives the entirety or portion of the image from the imagetransmission apparatus 1200 while the partial range of the image isselected (S1104).

Specifically, if the image belongs to the first type, the receiving unit1104 receives the entirety of the image from the image transmissionapparatus 1200. If the image belongs to the second type, the receivingunit 1104 receives the portion of the image from the image transmissionapparatus 1200. More specifically, if the image belongs to the secondtype, the receiving unit 1104 receives only one or more tilescorresponding to the partial range of the image among the plurality oftiles included in the image, each of the one or more tiles being a tileoutside which a reference to is prohibited in inter-picture prediction.

For example, a flag indicating whether the image belongs to the firsttype or the second type is attached to the image, and the receiving unit1104 may receive the entirety or portion of the image in accordance withthe flag.

Then, the decoding unit 1105 decodes the entirety or portion of theimage received by the receiving unit 1104 from the image transmissionapparatus 1200 (S1105).

Then, the display control unit 1106 displays the entirety or portion ofthe image decoded by the decoding unit 1105 on the display device 1107(S1106). The entirety or portion of the decoded image may be stored inthe saving unit 1110 as data. In this case, the display of the image maybe omitted.

Accordingly, the image transmission apparatus 1200 can switch betweentransmission of the entirety of an image and transmission of a portionof the image in accordance with the type of the image. The imagedecoding apparatus 1100 can also switch between decoding of the entiretyof an image and decoding of a portion of the image in accordance withthe type of the image. Thus, the image transmission apparatus 1200 andthe image decoding apparatus 1100 enable efficient processing of anencoded image.

Furthermore, for example, as described above, the range information maybe saved in the image decoding apparatus 1100, the image transmissionapparatus 1200, or the like. The receiving unit 1104 of the imagedecoding apparatus 1100 may receive a portion of the image in accordancewith the saved range information. Then, the decoding unit 1105 of theimage decoding apparatus 1100 may decode the portion of the imagereceived in accordance with the saved range information. Then, thedisplay control unit 1106 of the image decoding apparatus 1100 maydisplay the portion of the image received and decoded in accordance withthe saved range information.

For example, in a case where no range information is saved, thetransmission unit 1102 of the image decoding apparatus 1100 may transmitrange information to the image transmission apparatus 1200. In a casewhere range information is saved, the transmission unit 1102 of theimage decoding apparatus 1100 may not necessarily transmit rangeinformation to the image transmission apparatus 1200. In a case whererange information is saved and a portion of the image is to be received,the receiving unit 1104 of the image decoding apparatus 1100 may receivea portion of the image in accordance with the saved range informationwithout the range information being transmitted from the transmissionunit 1102 to the image transmission apparatus 1200.

For example, when a portion of the image is to be displayed after theentirety of another image has been displayed, the display control unit1106 may display a portion of the image received and decoded inaccordance with the saved range information. For example, in a casewhere display of a portion of the image is stopped and then resumed, thedisplay control unit 1106 may display a portion of the image receivedand decoded in accordance with the saved range information.

For example, the display control unit 1106 may display the portion ofthe image received and decoded in accordance with the saved rangeinformation on a display device different from the display device 1107.

Other Modifications

The configuration according to the present disclosure is not limited tothat in the embodiments described above. The configuration according tothe present disclosure may be in any of the following forms.

(1) Each of the apparatuses described above (i.e., each of the imagedecoding apparatus, the image transmission apparatus, and so forth) is,specifically, a computer system including a microprocessor, a ROM, arandom access memory (RAM), a hard disk unit, a display unit, akeyboard, a mouse, and so forth.

The RAM or the hard disk unit stores a computer program. Themicroprocessor operates in accordance with the computer program,allowing each apparatus to achieve its functions. The computer programincludes, in combination, a plurality of instruction codes eachindicating a command for a computer to achieve a predetermined function.

(2) Some or all of a plurality of constituent elements included in eachof the apparatuses described above may be formed of a single system LSI.A system LSI is an ultra multifunctional LSI fabricated by integrating aplurality of components onto a single chip, and is, specifically, acomputer system including a microprocessor, a ROM, a RAM, and so forth.

The RAM stores a computer program. The microprocessor operates inaccordance with the computer program, allowing the system LSI to achieveits functions.

(3) Some or all of a plurality of constituent elements included in eachof the apparatuses described above may also be formed of an integratedcircuit (IC) card attachable to each apparatus or formed of a singlemodule. The IC card or the module is a computer system including amicroprocessor, a ROM, a RAM, and so forth.

The IC card or the module includes the ultra multifunctional LSIdescribed above. The microprocessor operates in accordance with thecomputer program, allowing the IC card or the module to achieve itsfunctions. The IC card or the module may have tamper resistance.

(4) The configuration according to the present disclosure may providethe method described above. The configuration according to the presentdisclosure may provide a computer program according to which the methoddescribed above is implemented by a computer, or may be a digital signalincluding the computer program.

The configuration according to the present disclosure may also provide acomputer-readable recording medium having the computer program or thedigital signal recorded thereon. Examples of the computer-readablerecording medium include a flexible disk, a hard disk, a CD-ROM, amagneto-optical (MO) disk, a DVD, a DVD-ROM, a DVD-RAM, a BD, and asemiconductor memory. The configuration according to the presentdisclosure may also provide the digital signal recorded on the recordingmedium described above.

The configuration according to the present disclosure may also be aconfiguration in which the computer program or the digital signal istransmitted via a telecommunications line, a wireless communicationline, a wired communication line, a network such as the Internet, databroadcasting, or the like.

The configuration according to the present disclosure may also provide acomputer system including a microprocessor and a memory. The memory maystore the computer program described above, and the microprocessor mayoperate in accordance with the computer program.

The program or the digital signal may be recorded on a recording mediumand then the recording medium may be transferred, or the program or thedigital signal may be transferred via a network or the like. Theconfiguration according to the present disclosure may further beimplemented by another independent computer system.

(5) In the embodiments described above, an image decoding method, animage decoding apparatus, an image transmission method, an imagetransmission apparatus, and the like have been mainly described. It isto be understood that an image encoding method and an image encodingapparatus for generating a bit stream, which enable operation of theimage decoding method, the image decoding apparatus, the imagetransmission method, the image transmission apparatus, and the like, maybe used. The image encoding apparatus and the image transmissionapparatus may be the same. Some apparatuses or circuits that performpart of processing may be located in another country. As long as asystem according to the present disclosure is constructed as whole, thesystem falls within the scope of the present disclosure.

(6) The embodiments and the modification described above may be used incombination.

In each of the above-described embodiments, each constituent element maybe implemented by dedicated hardware, or may be implemented by theexecution of a software program suitable for the constituent element.Each constituent element may be implemented by a program execution unit,such as a central processing unit (CPU) or a processor, reading andexecuting a software program recorded on a recording medium such as ahard disk or a semiconductor memory.

In other words, an image decoding apparatus and an image transmissionapparatus includes a processing circuit (“processing circuitry”) and astorage device (“storage”) electrically connected to the processingcircuit (or accessible from the processing circuit). The processingcircuit includes at least one of dedicated hardware and a programexecution unit, and executes processing using the storage device. If theprocessing circuit includes a program execution unit, the storage devicestores a software program executed by the program execution unit.

Here, software that implements an image decoding apparatus or the likein each of the above-described embodiments is the following program.

The program causes a computer to execute an image decoding method forreceiving an encoded image from an image transmission apparatus anddecoding the image received from the image transmission apparatus. Theimage decoding method includes a requesting step of requesting the imagetransmission apparatus to transmit the image; a transmitting step oftransmitting range information to the image transmission apparatus, therange information being range information indicating a range which is apartial range of the image within an entire range of the image and whichis selected in accordance with a user operation; a receiving step of,while the partial range of the image is selected, (i) receiving theentirety of the image from the image transmission apparatus when theimage belongs to a first type, and (ii) receiving a portion of the imagefrom the image transmission apparatus when the image belongs to a secondtype, by receiving one or more tiles corresponding to the partial rangeof the image among a plurality of tiles included in the image, each ofthe one or more tiles being a tile outside which a reference to isprohibited in inter-picture prediction; and a decoding step of decodingthe entirety or portion of the image received in the receiving step fromthe image transmission apparatus.

The program may also be a program for causing a computer to execute animage transmission method for transmitting an encoded image to an imagedecoding apparatus. The image transmission method includes a requestaccepting step of accepting a request from the image decoding apparatusto transmit the image; a receiving step of receiving range informationfrom the image decoding apparatus, the range information being rangeinformation indicating a range which is a partial range of the imagewithin an entire range of the image and which is selected in accordancewith a user operation; and a transmitting step of, while the partialrange of the image is selected, (i) transmitting the entirety of theimage to the image decoding apparatus when the image belongs to a firsttype, and (ii) transmitting a portion of the image to the image decodingapparatus when the image belongs to a second type, by transmitting oneor more tiles corresponding to the partial range of the image among aplurality of tiles included in the image, each of the one or more tilesbeing a tile outside which a reference to is prohibited in inter-pictureprediction.

As described above, the individual constituent elements may be circuits.The circuits may form a single circuit in its entirety, or may beseparate circuits. Further, the individual constituent elements may beimplemented by a general-purpose processor, or may be implemented by adedicated processor.

In addition, a process executed by a specific constituent element may beexecuted by another constituent element. Further, the order in whichprocesses are executed may be changed, or a plurality of processes maybe executed in parallel. In addition, an image processing apparatus mayinclude an image decoding apparatus and an image transmission apparatus.

The image processing system, the image transmission apparatus, the imagetransmission method, the image decoding apparatus, and the imagedecoding method described above may be expressed as an imagedistribution system, an image distribution apparatus, an imagedistribution method, an image receiving apparatus, and an imagereceiving method, respectively.

While an image decoding apparatus and an image transmission apparatusaccording to one or a plurality of aspects have been described withreference to embodiments, the present disclosure is not limited to thoseembodiments. Various modifications conceivable by a person skilled inthe art and made to the embodiments described above or an embodiment inwhich constituent elements in different embodiments are used incombination may also fall within the scope of one or a plurality ofaspects without departing from the scope of the present disclosure.

Sixth Embodiment

A program for implementing the configuration of the moving imageencoding method (image encoding method) or the moving image decodingmethod (image decoding method) illustrated in each of theabove-described embodiments is recorded on a storage medium, enablingthe process illustrated in each of the above-described embodiments to beeasily implemented in an independent computer system. The storage mediummay be any medium capable of having a program recorded thereon, such asa magnetic disk, an optical disk, a magneto-optical disk, an IC card, ora semiconductor memory.

Hereinafter, application examples of the moving image encoding method(image encoding method) or the moving image decoding method (imagedecoding method) illustrated in each of the above-described embodimentsand a system using the same will further be described. The systemincludes an image encoding and decoding apparatus. The image encodingand decoding apparatus includes an image encoding apparatus that uses animage encoding method, and an image decoding apparatus that uses animage decoding method. Other configuration in the system may bemodified, as appropriate, depending on the situation.

FIG. 31 is a diagram illustrating the overall configuration of a contentsupply system ex100 that achieves the provision of a contentdistribution service. An area over which a communication service is tobe provided is divided into cells each having the desired size, and eachcell has a base station ex106, ex107, ex108, ex109, or ex110, which is afixed radio station, located therein.

In the content supply system ex100, devices such as a computer ex111, apersonal digital assistant (PDA) ex112, a camera ex113, a mobile phoneex114, and a game device ex115 are connected to the Internet ex101 viaan Internet service provider ex102, a telephone network ex104, and thebase stations ex106 to ex110.

However, the configuration of the content supply system ex100 is notlimited to that illustrated in FIG. 31, and one of the elementsdescribed above may be combined with and connected to another element.In addition, the individual devices may be connected directly to thetelephone network ex104 without the intervention of the base stationsex106 to ex110, each of which is a fixed radio station. Furthermore, theindividual devices may be connected directly to one another viashort-range radio or the like.

The camera ex113 is a device configured to capture a moving image, suchas a digital video camera, and a camera ex116 is a device configured tocapture a still image and a moving image, such as a digital camera.Examples of the mobile phone ex114 include a Global System for MobileCommunications (GSM (registered trademark)) mobile phone, a CodeDivision Multiple Access (CDMA) mobile phone, a Wideband-Code DivisionMultiple Access (W-CDMA) mobile phone, a Long Term Evolution (LTE)mobile phone, a High Speed packet Access (HSPA) mobile phone, and aPersonal Handyphone System (PHS), any of which may be used.

In the content supply system ex100, for example, the camera ex113 isconnected to a streaming server ex103 via the base station ex109 and thetelephone network ex104, enabling live distribution and the like. Inlive distribution, a user captures an image using the camera ex113 tocreate content (such as video of live music), and the camera ex113performs an encoding process on the created content in the way describedin each of the above-described embodiments (that is, serving as an imageencoding apparatus according to an aspect of the present disclosure),and transmits the resulting content to the streaming server ex103. Thestreaming server ex103 distributes, as a stream, the transmitted contentdata to a client that has made a request. Examples of the client includethe computer ex111, the PDA ex112, the camera ex113, the mobile phoneex114, and the game device ex115, which are configured to decode datasubjected to the encoding process described above. Upon receipt of thedistributed data, each device performs a decoding process on thereceived data and reproduces the decoded data (that is, serving as animage decoding apparatus according to an aspect of the presentdisclosure).

The encoding process of the captured data may be performed by the cameraex113 or the streaming server ex103 that performs a data transmissionprocess, or may be shared and performed by the camera ex113 and thestreaming server ex103. Likewise, the decoding process of thedistributed data may be performed by the client or the streaming serverex103, or may be shared and performed by the client and the streamingserver ex103. In addition, still image data and/or moving image dataobtained by capturing an image using the camera ex116 instead of thecamera ex113 may be transmitted to the streaming server ex103 via thecomputer ex111. In this case, the encoding process of the still imagedata and/or the moving image data may be performed by any of the cameraex116, the computer ex111, and the streaming server ex103, or may beshared and performed by the camera ex116, the computer ex111, and thestreaming server ex103.

The encoding/decoding process described above is generally performed inthe computer ex111 or in an LSI ex500 included in each device. The LSIex500 may be a single chip or may be formed of a plurality of chips. Theencoding/decoding process may be performed by using software forencoding/decoding moving images that is incorporated into a recordingmedium (such as a CD-ROM, a flexible disk, or a hard disk) readable bythe computer ex111 or the like. If the mobile phone ex114 is equippedwith a camera, moving image data acquired by the camera may betransmitted. In this case, the moving image data is data encoded by theLSI ex500 included in the mobile phone ex114.

In addition, the streaming server ex103 may be implemented by aplurality of servers or a plurality of computers configured to process,record, and distribute data in a distributed manner.

In the manner described above, in the content supply system ex100, theencoded data can be received and reproduced by the client. In thecontent supply system ex100, accordingly, information transmitted by auser can be received, decoded, and reproduced by the client in realtime. Even a user who has no special rights or facilities is able toachieve personal broadcasting.

The example of the content supply system ex100 is not intended to belimiting. As illustrated in FIG. 32, at least one of a moving imageencoding apparatus (image encoding apparatus) or a moving image decodingapparatus (image decoding apparatus) in each of the above-describedembodiments may be incorporated into a digital broadcasting systemex200. Specifically, in a broadcast station ex201, multiplexed dataobtained by multiplexing music data or the like onto video data istransmitted to a communication or broadcast satellite ex202 via a radiowave. The video data is data encoded using the moving image encodingmethod described in each of the above-described embodiments (that is,data encoded by an image encoding apparatus according to an aspect ofthe present disclosure). Upon receipt of the multiplexed data, thebroadcast satellite ex202 sends a broadcasting radio wave, and a homeantenna ex204 capable of receiving satellite broadcasts receives theradio wave. The received multiplexed data is decoded and reproduced byan apparatus such as a TV (receiver) ex300 or a set-top box (STB) ex217(that is, the apparatus serves as an image decoding apparatus accordingto an aspect of the present disclosure).

In addition, the moving image decoding apparatus or the moving imageencoding apparatus illustrated in each of the above-describedembodiments may also be mounted in a reader/recorder ex218. Thereader/recorder ex218 reads multiplexed data recorded on a recordingmedium ex215 such as a DVD or a BD and decodes the read multiplexeddata, or encodes a video signal and further, depending on the situation,multiplexes the video signal with a music signal, and writes theresulting signal to the recording medium ex215. In this case, thereproduced video signal is displayed on a monitor ex219, and anotherapparatus or system can reproduce the video signal from the recordingmedium ex215 having the multiplexed data recorded thereon. In addition,the moving image decoding apparatus may be mounted in a set-top boxex217 connected to a cable TV cable ex203 or the antenna ex204 forsatellite/terrestrial broadcasting, and a decoded moving image may bedisplayed on the monitor ex219 of the TV ex300. The moving imagedecoding apparatus may be incorporated in the TV ex300 instead of theset-top box ex217.

FIG. 33 is a diagram illustrating a TV (receiver) ex300 that uses themoving image decoding method and the moving image encoding methoddescribed in each of the above-described embodiments. The TV ex300includes a tuner ex301, a modulation/demodulation unit ex302, and amultiplexing/demultiplexing unit ex303. The tuner ex301 acquires oroutputs multiplexed data obtained by multiplexing audio data onto videodata via the antenna ex204 that receives the broadcast described aboveor via the cable ex203 or the like. The modulation/demodulation unitex302 demodulates the received multiplexed data, or modulatesmultiplexed data to be transmitted to outside. Themultiplexing/demultiplexing unit ex303 separates the demodulatedmultiplexed data into video data and audio data, or multiplexes videodata and audio data that are encoded by a signal processing unit ex306.

The TV ex300 further includes the signal processing unit ex306 and anoutput unit ex309. The signal processing unit ex306 includes an audiosignal processing unit ex304 and a video signal processing unit ex305(serving as an image encoding apparatus or an image decoding apparatusaccording to an aspect of the present disclosure) that decodes audiodata and video data, respectively, or encodes respective information.The output unit ex309 includes a speaker ex307 that outputs the decodedaudio signal, and a display unit ex308, such as a display, that displaysthe decoded video signal. The TV ex300 further includes an interfaceunit ex317. The interface unit ex317 includes, for example, an operationinput unit ex312 that accepts input of a user operation. The TV ex300further includes a control unit ex310 for overall control of theindividual units, and a power supply circuit unit ex311 for supplyingpower to each of the individual units. The interface unit ex317 mayinclude components other than the operation input unit ex312, examplesof which include a bridge ex313 to be connected to an external devicesuch as the reader/recorder ex218, a slot unit ex314 into which arecording medium ex216 such as an SD card is removably placed, a driverex315 to be connected to an external recording medium such as a harddisk, and a modem ex316 to be connected to the telephone network ex104.The recording medium ex216 has a non-volatile/volatile semiconductormemory element capable of electrically recording information. Theindividual units of the TV ex300 are connected to one another via asynchronization bus.

First, a description will be given of a configuration in which the TVex300 decodes and reproduces multiplexed data acquired from outside viathe antenna ex204 or the like. In the TV ex300, in response to a useroperation from a remote controller ex220 or the like, themultiplexing/demultiplexing unit ex303 separates the multiplexed datademodulated by the modulation/demodulation unit ex302 under control ofthe control unit ex310, which has a CPU and the like. In the TV ex300,furthermore, the separated audio data and the separated video data aredecoded by the audio signal processing unit ex304 and the video signalprocessing unit ex305, respectively, using the decoding method describedin each of the above-described embodiments. The decoded audio signal andvideo signal are output to outside from the output unit ex309. When theaudio signal and the video signal are to be output, the audio signal andthe video signal may be temporarily accumulated in buffers ex318 andex319 or the like so that the audio signal and the video signal aresynchronously reproduced. In the TV ex300, furthermore, multiplexed datamay be read from the recording medium ex215 or ex216, such as amagnetic/optical disk or an SD card, rather than from a broadcast or thelike. A description will now be given of a configuration in which the TVex300 encodes an audio signal and a video signal and transmits theencoded audio signal and video signal to outside or writes the encodedaudio signal and video signal to a recording medium or the like. In theTV ex300, in response to a user operation from the remote controllerex220 or the like, the audio signal processing unit ex304 and the videosignal processing unit ex305 encode an audio signal and a video signal,respectively, using the encoding method described in each of theabove-described embodiments under control of the control unit ex310. Theencoded audio signal and video signal are multiplexed by themultiplexing/demultiplexing unit ex303, and are output to the outside.When the audio signal and the video signal are to be multiplexed, theaudio signal and the video signal may be temporarily accumulated inbuffers ex320 and ex321 or the like so that the audio signal and thevideo signal are synchronized with each other. A plurality of buffers,such as the buffers ex318, ex319, ex320, and ex321, as illustrated inFIG. 33 may be used, or one or more buffers may be shared. In additionto those illustrated in FIG. 33, data may be accumulated in a bufferserving as a buffering mechanism to prevent system overflow andunderflow between, for example, the modulation/demodulation unit ex302and the multiplexing/demultiplexing unit ex303.

In addition, the TV ex300 may have a configuration to accept audio/video(AV) input of a microphone or a camera, in addition to acquiring audiodata and video data from a broadcast, a recording medium, or the like,and may perform an encoding process on acquired data. While the TV ex300has been described as having a configuration capable of the encodingprocess, multiplexing process, and external output process describedabove, the TV ex300 may be configured not to be capable of theabove-described processes and may have a configuration capable of onlythe receiving process, decoding process, and external output processdescribed above.

When the reader/recorder ex218 reads or writes multiplexed data from arecording medium, the decoding process or encoding process describedabove may be performed by either of the TV ex300 and the reader/recorderex218, or may be shared and performed by the TV ex300 and thereader/recorder ex218.

FIG. 34 illustrates, by way of example, the configuration of aninformation reproducing/recording unit ex400 for reading or writing datafrom or to an optical disk. The information reproducing/recording unitex400 includes an optical head ex401, a modulation and recording unitex402, a reproduction and demodulation unit ex403, a buffer ex404, adisk motor ex405, a servo control unit ex406, and a system control unitex407. The optical head ex401 applies a laser spot to a recordingsurface of the recording medium ex215, which is an optical disk, towrite information, and detects light reflected from the recordingsurface of the recording medium ex215 to read the information. Themodulation and recording unit ex402 electrically drives a semiconductorlaser in the optical head ex401, and modulates a laser beam inaccordance with the recording data. The reproduction and demodulationunit ex403 amplifies a reproduction signal produced by electricallydetecting light reflected from the recording surface using aphotodetector in the optical head ex401, and separates signal componentsrecorded on the recording medium ex215 for demodulation to reproducenecessary information. The buffer ex404 temporarily holds information tobe recorded on the recording medium ex215 and information reproducedfrom the recording medium ex215. The disk motor ex405 rotates therecording medium ex215. The servo control unit ex406 causes the opticalhead ex401 to move to a predetermined information track whilecontrolling the driving of the disk motor ex405 to rotate, and tracksthe laser spot. The system control unit ex407 provides overall controlof the information reproducing/recording unit ex400. The reading andwriting process described above is implemented by the system controlunit ex407 by using various information held in the buffer ex404 and, ifnecessary, by generating and adding new information and by recording andreproducing information through the optical head ex401 while causing themodulation and recording unit ex402, the reproduction and demodulationunit ex403, and the servo control unit ex406 to operate in cooperationwith one another. The system control unit ex407 is composed of, forexample, a microprocessor, and executes a reading and writing program toexecute the reading and writing process.

In the foregoing description, the optical head ex401 applies a laserspot. The optical head ex401 may be configured to perform higher densityrecording with near-field light.

FIG. 35 is a schematic diagram of the recording medium ex215, which isan optical disk. The recording surface of the recording medium ex215 hasa guide groove (or groove) formed in a spiral shape. An informationtrack ex230 has recorded thereon in advance address information whichindicates an absolute position on the disk using a change in the shapeof the groove. The address information includes information foridentifying the position of a recording block ex231 that is the unit forwhich data is recorded. An apparatus used for recording or reproductionreproduces the information track ex230 and reads the address informationto identify a recording block. The recording medium ex215 also has adata recording area ex233, an inner circumference area ex232, and anouter circumference area ex234. The data recording area ex233 is an areaused to record user data. The inner circumference area ex232 and theouter circumference area ex234 are located inside and outside the datarecording area ex233, respectively, and are used for specific purposesother than recording of user data. The information reproducing/recordingunit ex400 reads and writes encoded audio data and video data ormultiplexed data obtained by multiplexing the audio data and video datafrom and to the data recording area ex233 of the recording medium ex215having the configuration described above.

In the foregoing description, by way of example, an optical disk such asa single-layer DVD or BD is used. The optical disk is merely an example,and an optical disk having a multi-layer structure and capable ofrecording on multiple layers may be used. Alternatively, an optical diskhaving a structure for multidimensional recording/reproduction, such asrecording information in the same location on the disk by using light ofdifferent colors having different wavelengths or recording differentlayers of information from different angles, may be used.

In the digital broadcasting system ex200, furthermore, a vehicle ex210having an antenna ex205 may be able to receive data from the satelliteex202 or the like and to reproduce a moving image on a display device ofa car navigation system ex211 or the like included in the vehicle ex210.The car navigation system ex211 may have a configuration in which, forexample, a GPS receiving unit is added to the configuration illustratedin FIG. 33, and the computer ex111, the mobile phone ex114, and the likemay also have a similar configuration.

FIG. 36A is a diagram illustrating the mobile phone ex114 that uses themoving image decoding method and the moving image encoding methoddescribed in the embodiments described above. The mobile phone ex114includes an antenna ex350 for transmitting and receiving radio waves toand from the base station ex110, a camera unit ex365 configured tocapture video and a still image, and a display unit ex358, such as aliquid crystal display (LCD), that displays data obtained by decodingthe video captured by the camera unit ex365, video received by theantenna ex350, or the like. The mobile phone ex114 further includes amain body having an operation key unit ex366, an audio output unitex357, such as a speaker, for outputting audio, an audio input unitex356, such as a microphone, for inputting audio, and a memory unitex367 that saves encoded data or decoded data for captured video orstill images, recorded audio, received video, still images, or email, orthe like, or a slot unit ex364 serving as an interface unit thatinterfaces with a recording medium that saves data in a similar way.

An example configuration of the mobile phone ex114 will further bedescribed with reference to FIG. 36B. In the mobile phone ex114, a powersupply circuit unit ex361, an operation input control unit ex362, avideo signal processing unit ex355, a camera interface unit ex363, anLCD control unit ex359, a modulation/demodulation unit ex352, amultiplexing/demultiplexing unit ex353, an audio signal processing unitex354, the slot unit ex364, and the memory unit ex367 are connected to amain control unit ex360 via a bus ex370. The main control unit ex360provides overall control of the display unit ex358 and the individualunits of the main body including the operation key unit ex366.

When a call is terminated and a power key is turned on by a useroperation, the power supply circuit unit ex361 supplies power from abattery pack to each unit to activate the mobile phone ex114 intooperation.

In the mobile phone ex114, an audio signal obtained by collecting audiousing the audio input unit ex356 in a voice communication mode isconverted into a digital audio signal using the audio signal processingunit ex354, subjected to spread spectrum processing using themodulation/demodulation unit ex352, subjected to digital-to-analogconversion processing and frequency transform processing using atransmitting/receiving unit ex351, and then transmitted via the antennaex350 under control of the main control unit ex360, which has a CPU, aROM, a RAM, and so forth. In the mobile phone ex114, furthermore,received data received via the antenna ex350 in the voice communicationmode is amplified, subjected to frequency transform processing andanalog-to-digital conversion processing, subjected to inverse spreadspectrum processing using the modulation/demodulation unit ex352,converted into an analog audio signal using the audio signal processingunit ex354, and then output from the audio output unit ex357.

Further, in a case where electronic mail is to be transmitted in a datacommunication mode, text data of electronic mail input by the operationof the operation key unit ex366 or the like of the main body isdelivered to the main control unit ex360 via the operation input controlunit ex362. Under control of the main control unit ex360, the text datais subjected to spread spectrum processing using themodulation/demodulation unit ex352, subjected to digital-to-analogconversion processing and frequency transform processing using thetransmitting/receiving unit ex351, and then transmitted to the basestation ex110 via the antenna ex350. In a case where electronic mail isreceived, the received data is subjected to processing substantiallyinverse to the processing described above, and the result is output tothe display unit ex358.

In a case where video, a still image, or video and audio are to betransmitted in the data communication mode, the video signal processingunit ex355 compresses and encodes the video signal supplied from thecamera unit ex365 using the moving image encoding method illustrated ineach of the above-described embodiments (that is, serving as an imageencoding apparatus according to an aspect of the present disclosure),and delivers the encoded video data to the multiplexing/demultiplexingunit ex353. Further, the audio signal processing unit ex354 encodes anaudio signal obtained by collecting audio using the audio input unitex356 for a period during which the camera unit ex365 captures video, astill image, or the like, and delivers the encoded audio data to themultiplexing/demultiplexing unit ex353.

The multiplexing/demultiplexing unit ex353 multiplexes the encoded videodata supplied from the video signal processing unit ex355 and theencoded audio data supplied from the audio signal processing unit ex354using a predetermined scheme. The resulting multiplexed data issubjected to spread spectrum processing using themodulation/demodulation unit (or modulation/demodulation circuit unit)ex352, subjected to digital-to-analog conversion processing andfrequency transform processing using the transmitting/receiving unitex351, and then transmitted via the antenna ex350.

In a case where data of a moving image file linked to a homepage or thelike is received in the data communication mode or in a case whereelectronic mail with video and/or audio attached thereto is received, inorder to decode the multiplexed data received via the antenna ex350, themultiplexing/demultiplexing unit ex353 separates the multiplexed datainto a bit stream of video data and a bit stream of audio data, andsupplies the encoded video data to the video signal processing unitex355 and the encoded audio data to the audio signal processing unitex354 via a synchronization bus ex370. The video signal processing unitex355 decodes the video signal using the moving image decoding methodcorresponding to the moving image encoding method illustrated in each ofthe above-described embodiments (that is, serving as an image decodingapparatus according to an aspect of the present disclosure), and videoor a still image included in, for example, the moving image file linkedto the homepage is displayed from the display unit ex358 via the LCDcontrol unit ex359. The audio signal processing unit ex354 decodes theaudio signal, and audio is output from the audio output unit ex357.

Furthermore, similarly to the TV ex300, terminals such as the mobilephone ex114 may have three implementation types: (1) a transmit- andreceive-type terminal including an encoder and a decoder, (2) a transmitterminal including an encoder, and (3) a receive terminal including adecoder. In the foregoing description, multiplexed data obtained bymultiplexing music data or the like onto video data is received andtransmitted in the digital broadcasting system ex200. The multiplexeddata may also be data in which data other than audio data, such as textdata related to video, is multiplexed. Alternatively, video data insteadof multiplexed data may be used.

In the manner described above, the moving image encoding method or themoving image decoding method illustrated in each of the above-describedembodiments may be used in any of the devices and systems describedabove, in which case the advantages described in each of theabove-described embodiments are achievable.

In addition, the present disclosure is not limited to the embodimentsdescribed above, and a variety of modifications or alterations can bemade without departing from the scope of the present disclosure.

Seventh Embodiment

Video data may also be generated by appropriately switching between themoving image encoding method or apparatus illustrated in each of theabove-described embodiments and a moving image encoding method orapparatus complying with different standards such as MPEG-2, MPEG4-AVC,and VC-1, if necessary.

If a plurality of pieces of video data each complying with a differentstandard are generated, decoding of the pieces of video data involvesselecting decoding methods corresponding to the respective standards.However, since it is difficult to identify which standard the video datato be decoded complies with, it is difficult to select an appropriatedecoding method.

To address this problem, multiplexed data obtained by multiplexing audiodata or the like onto video data includes identification informationindicating which standard the video data complies with. A specificconfiguration of multiplexed data including video data generated by themoving image encoding method or apparatus illustrated in each of theabove-described embodiments will be described hereinafter. Themultiplexed data is a digital stream having an MPEG-2 transport streamformat.

FIG. 37 is a diagram illustrating the configuration of the multiplexeddata. As illustrated in FIG. 37, the multiplexed data is obtained bymultiplexing one or more of a video stream, an audio stream, apresentation graphics (PG) stream, and an interactive graphics (IG)stream. The video stream indicates main video and sub-video in a movie.The audio stream indicates main audio in the movie and sub-audio to bemixed with the main audio. The presentation graphics stream includessubtitles in the movie. The main video indicates normal video displayedon a screen, and the sub-video is video displayed on a small screenwithin the main video. The interactive graphics stream indicates aninteractive screen created by placing a graphical user interface (GUI)component on the screen. The video stream is encoded by the moving imageencoding method or apparatus illustrated in each of the above-describedembodiments, or by a moving image encoding method or apparatus complyingwith an existing standard such as MPEG-2, MPEG4-AVC, or VC-1. The audiostream is encoded with a scheme such as Dolby AC-3, Dolby Digital Plus,Meridian Lossless Packing (MLP), Digital Theatre Systems (DTS), DigitalTheatre Sound High Definition (DTS-HD), or linear Pulse Code Modulation(PCM).

Each stream included in the multiplexed data is identified using apacket identifier (PID). For example, a video stream used for video of amovie is assigned 0x1011, audio streams are assigned 0x1100 to 0x111F,presentation graphics streams are assigned 0x1200 to 0x121F, interactivegraphics streams are assigned 0x1400 to 0x141F, video streams used forsub-video of the movie are assigned 0x1B00 to 0x1B1F, and audio streamsused for sub-audio to be mixed with the main audio are assigned 0x1A00to 0x1A1F.

FIG. 38 is a diagram schematically illustrating how multiplexed data ismultiplexed. First, a video stream ex235 including a plurality of videoframes and an audio stream ex238 including a plurality of audio framesare converted into packetized elementary stream (PES) packet sequencesex236 and ex239, respectively, and are further converted into transportstream (TS) packets ex237 and ex240, respectively. Likewise, data of apresentation graphics stream ex241 and data of an interactive graphicsstream ex244 are converted into PES packet sequences ex242 and ex245,respectively, and are further converted into TS packets ex243 and ex246,respectively. The TS packets ex237, ex240, ex243, and ex246 aremultiplexed into a single stream to produce multiplexed data ex247.

FIG. 39 illustrates in more detail how a video stream is stored in a PESpacket sequence. The first row of FIG. 39 illustrates a video framesequence of a video stream. The second row illustrates a PES packetsequence. As indicated by arrows yy1, yy2, yy3, and yy4 in FIG. 39, an Ipicture, B pictures, and a P picture, which are a plurality of VideoPresentation Units in the video stream, are separated on apicture-by-picture basis, and are stored in the payloads of PES packets.Each PES packet has a PES header, and the PES header contains apresentation time stamp (PTS) indicating the time when the picture isdisplayed and a decoding time stamp (DTS) indicating the time when thepicture is decoded.

FIG. 40 illustrates the format of TS packets finally written inmultiplexed data. Each TS packet is a 188-byte fixed-length packethaving a 4-byte TS header containing information for identifying thestream, such as a PID, and a 184-byte TS payload containing data. ThePES packet described above is divided and is stored in a TS payload. Inthe case of a BD-ROM, a TS packet is assigned 4-byte TP_Extra_Header,thereby forming a 192-byte source packet, which is written inmultiplexed data. TP_Extra_Header contains information such as an ATS(Arrival_Time_Stamp). The ATS indicates the time when the TS packet isstarted to be transferred to a PID filter of a decoder. As illustratedin the bottom row of FIG. 40, the multiplexed data has a sequence ofsource packets whose numbers with increments starting from the beginningof the multiplexed data are referred to as source packet numbers (SPNs).

Further, the TS packets included in the multiplexed data include, inaddition to the streams of video, audio, subtitles, and so forth, aprogram association table (PAT), a program map table (PMT), a programclock reference (PCR), and so forth. The PAT indicates what the PID ofthe PMT used in the multiplexed data is, and the PAT's own PID isregistered as 0. The PMT has PIDs of streams of video, audio, subtitles,and so forth included in the multiplexed data, and attribute informationon the streams corresponding to the respective PIDs. The PMT alsoincludes various descriptors concerning the multiplexed data. Thedescriptors include copy control information indicating whether copyingof the multiplexed data is permitted or not. The PCR has information onthe system time clock (STC) time corresponding to the ATS at which thePCR packet is transferred to the decoder to achieve synchronizationbetween an arrival time clock (ATC), which is the ATS time axis, and anSTC, which is the PTS/DTS time axis.

FIG. 41 is a diagram explaining the data structure of the PMT in detail.The PMT has a PMT header placed at the top thereof. The PMT headercontains the length of the data included in the PMT, and so forth. ThePMT header is followed by a plurality of descriptors concerning themultiplexed data. The copy control information described above and thelike are written as descriptors. The descriptors are followed by aplurality of pieces of stream information concerning the individualstreams included in the multiplexed data. Each piece of streaminformation is composed of a stream type for identifying the compressioncodec and the like of the stream, the PID of the stream, and streamdescriptors containing attribute information on the stream (such as theframe rate and the aspect ratio). The number of stream descriptors isequal to the number of streams present in the multiplexed data.

When recorded on a recording medium or the like, the multiplexed data isrecorded together with a multiplexed data information file.

As illustrated in FIG. 42, the multiplexed data information fileincludes management information on the multiplexed data. Multiplexeddata information files and pieces of multiplexed data are in one-to-onecorrespondence with each other. Each multiplexed data information fileis composed of multiplexed data information, stream attributeinformation, and entry map.

As illustrated in FIG. 42, the multiplexed data information is composedof a system rate, a reproduction start time, and a reproduction endtime. The system rate indicates the maximum rate at which themultiplexed data is transferred to a PID filter of a system targetdecoder described below. The interval between ATSs included in themultiplexed data is set to be less than or equal to the system rate. Thereproduction start time is the PTS of the first video frame of themultiplexed data, and the reproduction end time is set by adding thereproduction interval corresponding to one frame to the PTS of the lastvideo frame of the multiplexed data.

In the stream attribute information, as illustrated in FIG. 43,attribute information regarding each stream included in the multiplexeddata is registered for each PID. The attribute information has differentinformation for each of the video stream, the audio stream, thepresentation graphics stream, and the interactive graphics stream. Thevideo stream attribute information has information such as thecompression codec with which the video stream has been compressed, andthe resolution, aspect ratio, and frame rate of each piece of picturedata included in the video stream. The audio stream attributeinformation has information such as the compression codec with which theaudio stream has been compressed, the number of channels included in theaudio stream, the language of the audio stream, and the samplingfrequency. The above-described pieces of information are used for, forexample, the initialization of the decoder before a player reproducesthe multiplexed data.

In this embodiment, the stream type included in the PMT within themultiplexed data is used. If the multiplexed data is recorded on arecording medium, the video stream attribute information included in themultiplexed data information is used. Specifically, the moving imageencoding method or apparatus illustrated in each of the above-describedembodiments includes a step or unit for setting unique information,which indicates video data generated by the moving image encoding methodor apparatus illustrated in each of the above-described embodiments, inthe video stream attribute information or in the stream type included inthe PMT. The configuration described above allows video data generatedby the moving image encoding method or apparatus illustrated in each ofthe above-described embodiments and video data complying with otherstandards to be distinguished from each other.

FIG. 44 illustrates the steps of a moving image decoding methodaccording to this embodiment. In step exS100, the stream type includedin the PMT or the video stream attribute information included in themultiplexed data information is acquired from the multiplexed data.Then, in step exS101, it is determined whether or not the stream type orthe video stream attribute information indicates that the multiplexeddata is multiplexed data generated by the moving image encoding methodor apparatus illustrated in each of the above-described embodiments. Ifit is determined that the stream type or the video stream attributeinformation indicates that the multiplexed data is multiplexed datagenerated by the moving image encoding method or apparatus illustratedin each of the above-described embodiments, then in step exS102, themultiplexed data is decoded using the moving image decoding methodillustrated in each of the above-described embodiments. If the streamtype or the video stream attribute information indicates that themultiplexed data complies with an existing standard such as MPEG-2,MPEG4-AVC, or VC-1, then in step exS103, the multiplexed data is decodedusing a moving image decoding method complying with the existingstandard.

In the manner described above, setting a new unique value in the streamtype or the video stream attribute information allows determination asto whether or not data is decodable by the moving image decoding methodor apparatus illustrated in each of the above-described embodiments whenthe data is decoded. Accordingly, even if multiplexed data complyingwith a different standard is input, an appropriate decoding method orapparatus can be selected, and thus the multiplexed data can be decodedwithout any error. In addition, the moving image encoding method orapparatus or the moving image decoding method or apparatus illustratedin this embodiment can also be used for any of the apparatuses andsystems described above.

Eighth Embodiment

The moving image encoding method and apparatus and the moving imagedecoding method and apparatus illustrated in each of the above-describedembodiments are implemented by an LSI, which is typically an integratedcircuit. FIG. 45 illustrates, by way of example, the configuration of anLSI ex500 fabricated into a single chip. The LSI ex500 includes acontrol unit ex501, a CPU ex502, a memory controller ex503, a streamcontroller ex504, a power supply circuit unit ex505, a streaminput/output unit (I/O) ex506, a signal processing unit ex507, a bufferex508, an AV I/O ex509, and driving frequency control unit ex512, whichare connected to one another via a bus ex510. The power supply circuitunit ex505 supplies power to each unit when power is turned on toactivate the LSI ex500 into operation.

For example, when an encoding process is to be performed, the LSI ex500receives, as input, an AV signal from the microphone ex117, the cameraex113, or the like via the AV I/O ex509 under control of the controlunit ex501, which includes the CPU ex502, the memory controller ex503,the stream controller ex504, the driving frequency control unit ex512,and so forth. The input AV signal is temporarily accumulated in anexternal memory ex511, such as a synchronous dynamic RAM (SDRAM). Theaccumulated data is delivered to the signal processing unit ex507, forexample, a plurality of times, if necessary, in accordance with theamount of processing or the processing speed under control of thecontrol unit ex501, and an audio signal is encoded and/or a video signalis encoded in the signal processing unit ex507. The encoding process ofthe video signal is the encoding process described in each of theabove-described embodiments. The signal processing unit ex507 furtherperforms processing, depending on the situation, such as multiplexingthe encoded audio data and the encoded video data, and outputs themultiplexed data to outside from the stream I/O ex506. The outputmultiplexed data is transmitted to the base station ex107, or is writtenin the recording medium ex215. When data is to be multiplexed, it isdesirable that the data be temporarily accumulated in the buffer ex508for synchronization.

In the foregoing description, the memory ex511 is configured to belocated outside the LSI ex500. The memory ex511 may be contained in theLSI ex500. The number of buffers ex508 is not limited to one, and aplurality of buffers may be used. Furthermore, the LSI ex500 may befabricated into a single chip, or may be separated into a plurality ofchips.

In the foregoing, furthermore, the control unit ex501 includes the CPUex502, the memory controller ex503, the stream controller ex504, thedriving frequency control unit ex512, and so forth. The control unitex501 may not necessarily have the configuration described above. Forexample, the signal processing unit ex507 may further include a CPU. Anadditional CPU included in the signal processing unit ex507 can furtherincrease the processing speed. In another example, the CPU ex502 mayinclude the signal processing unit ex507, or may include part of thesignal processing unit ex507, for example, an audio signal processingunit. In this case, the control unit ex501 has a configuration includingthe CPU ex502 having the signal processing unit ex507 or having part ofthe signal processing unit ex507.

While an LSI is termed here, any other term such as an IC, a system LSI,a super LSI, or an ultra LSI may be applicable in accordance with thedifference in the degree of integration.

The approach of fabricating an integrated circuit is not limited to LSI,and may be implemented by a dedicated circuit or a general-purposeprocessor. A field programmable gate array (FPGA) that is programmableafter an LSI is fabricated or a reconfigurable processor capable ofreconfiguring the connection or setting of circuit cells in the LSI maybe used. Such a programmable logic device is capable of executing themoving image encoding method or the moving image decoding methodillustrated in each of the above-described embodiments by typicallyloading a program constituting software or firmware or reading theprogram from a memory or the like.

Furthermore, in the case of the advent of integrated circuit technologyreplacing LSI due to the advancement of semiconductor technology orderivative other technology, it is to be understood that functionalblocks may be integrated by using this technology. Applications ofbiotechnology may be possible.

Ninth Embodiment

In a case where video data generated by the moving image encoding methodor apparatus illustrated in each of the above-described embodiments isdecoded, the amount of processing may be larger than that of video datacomplying with an existing standard such as Moving Picture ExpertsGroup-2 (MPEG-2), MPEG4-AVC, or VC-1. Thus, it is desirable that the LSIex500 set the driving frequency to a higher frequency than the drivingfrequency of the CPU ex502 when video data complying with an existingstandard is decoded. However, if the driving frequency is increased,power consumption will increase.

To address this problem, a moving image decoding apparatus such as theTV ex300 or the LSI ex500 is configured to identify which standard thevideo data complies with and to switch the driving frequency inaccordance with the standard. FIG. 46 illustrates a configuration ex800in this embodiment. If the video data is generated by the moving imageencoding method or apparatus illustrated in each of the above-describedembodiments, a driving frequency switching unit ex803 sets the drivingfrequency to a high frequency. Then, the driving frequency switchingunit ex803 instructs a decoding processing unit ex801 that executes themoving image decoding method illustrated in each of the above-describedembodiments to decode the video data. On the other hand, if the videodata is video data complying with an existing standard, the drivingfrequency switching unit ex803 sets the driving frequency to a lowerfrequency than that when the video data is video data generated by themoving image encoding method or apparatus illustrated in each of theabove-described embodiments. Then, the driving frequency switching unitex803 instructs a decoding processing unit ex802 compatible with theexisting standard to decode the video data.

More specifically, the driving frequency switching unit ex803 includesthe CPU ex502 and the driving frequency control unit ex512 illustratedin FIG. 45. In addition, the decoding processing unit ex801 thatexecutes the moving image decoding method illustrated in each of theabove-described embodiments and the decoding processing unit ex802compatible with an existing standard correspond to the signal processingunit ex507 illustrated in FIG. 45. The CPU ex502 identifies whichstandard the video data complies with. In accordance with a signal fromthe CPU ex502, the driving frequency control unit ex512 sets the drivingfrequency. In addition, in accordance with a signal from the CPU ex502,the signal processing unit ex507 decodes the video data. The video datamay be identified by using, for example, the identification informationdescribed in the seventh embodiment. The identification information isnot limited to that described in the seventh embodiment, and anyinformation capable of identifying which standard the video datacomplies with may be used. For example, in a case where it is possibleto identify which standard the video data complies with in accordancewith an external signal for identifying whether the video data isavailable for a TV or is available for a disk or the like, the videodata may be identified in accordance with the external signal. Inaddition, the CPU ex502 may select a driving frequency in accordancewith, for example, a look-up table illustrated in FIG. 48 in which videodata standards and driving frequencies are associated with each other.The look-up table may be stored in the buffer ex508 or in an internalmemory of an LSI, so that the CPU ex502 can refer to the look-up tableto select a driving frequency.

FIG. 47 illustrates steps for implementing a method according to thisembodiment. First, in step exS200, the signal processing unit ex507acquires identification information from multiplexed data. Then, in stepexS201, the CPU ex502 identifies, in accordance with the identificationinformation, whether or not the video data is data generated by theencoding method or apparatus illustrated in each of the above-describedembodiments. If the video data is data generated by the encoding methodor apparatus illustrated in each of the above-described embodiments,then in step exS202, the CPU ex502 delivers a signal for setting thedriving frequency to a high frequency to the driving frequency controlunit ex512. Then, the driving frequency control unit ex512 sets a highdriving frequency. On the other hand, if it is indicated that the videodata is video data complying with an existing standard such as MPEG-2,MPEG4-AVC, or VC-1, in step exS203, the CPU ex502 delivers a signal forsetting the driving frequency to a low frequency to the drivingfrequency control unit ex512. Then, the driving frequency control unitex512 sets a lower driving frequency than that when the video data isdata generated by the encoding method or apparatus illustrated in eachof the above-described embodiments.

Furthermore, varying a voltage to be applied to the LSI ex500 or anapparatus including the LSI ex500 in association with the switching ofthe driving frequency can further increase the power saving effect. Forexample, when the driving frequency is set low, a voltage to be appliedto the LSI ex500 or an apparatus including the LSI ex500 may also be setlower than that when the driving frequency is set high.

In the method for setting a driving frequency, it may be sufficient toset the driving frequency to a high frequency when the amount ofprocessing for decoding is large, and to set the driving frequency to alow frequency when the amount of processing for decoding is small. Thesetting method described above is not intended to be limiting. Forexample, if the amount of processing for decoding video data complyingwith the MPEG4-AVC standard is larger than the amount of processing fordecoding video data generated by the moving image encoding method orapparatus illustrated in each of the above-described embodiments, thedriving frequency may be set in a manner opposite to that describedabove.

Furthermore, the method for setting a driving frequency is not limitedto a configuration in which the driving frequency is set low. Forexample, when the identification information indicates that the videodata is data generated by the moving image encoding method or apparatusillustrated in each of the above-described embodiments, a voltage to beapplied to the LSI ex500 or an apparatus including the LSI ex500 may beset high, whereas, when the identification information indicates thatthe video data is video data complying with an existing standard such asMPEG-2, MPEG4-AVC, or VC-1, a voltage to be applied to the LSI ex500 oran apparatus including the LSI ex500 may be set low. In another example,when the identification information indicates that the video data isdata generated by the moving image encoding method or apparatusillustrated in each of the above-described embodiments, the driving ofthe CPU ex502 may not be stopped, whereas, when the identificationinformation indicates that the video data is video data complying withan existing standard such as MPEG-2, MPEG4-AVC, or VC-1, the driving ofthe CPU ex502 may be temporarily stopped because there is processing tospare. Even when the identification information indicates that the videodata is data generated by the moving image encoding method or apparatusillustrated in each of the above-described embodiments, the driving ofthe CPU ex502 may be temporarily stopped if there is processing tospare. In this case, the period of time during which the driving of theCPU ex502 is stopped may be set shorter than that when theidentification information indicates that the video data is video datacomplying with an existing standard such as MPEG-2, MPEG4-AVC, or VC-1.

In the manner described above, switching the driving frequency inaccordance with the standard with which the video data complies canachieve power saving. Furthermore, if the LSI ex500 or an apparatusincluding the LSI ex500 is driven by using a battery, power saving canresult in an increase in the life of the battery.

Tenth Embodiment

In some cases, a plurality of pieces of video data complying withdifferent standards may be input to the devices and systems describedabove, such as a TV and a mobile phone. In order to decode such aplurality of pieces of video data complying with different standardswhich have been input, it is desirable that the signal processing unitex507 of the LSI ex500 support a plurality of standards. However, if thesignal processing unit ex507 that supports each standard is individuallyused, the circuit scale of the LSI ex500 may increase, resulting in anincrease in cost.

To address this problem, a decoding processing unit for executing themoving image decoding method illustrated in each of the above-describedembodiments and a decoding processing unit compatible with an existingstandard such as MPEG-2, MPEG4-AVC, or VC-1 are configured to be sharedin part. FIG. 49A illustrates an example configuration ex900. Forexample, some processes are partially common to the moving imagedecoding method illustrated in each of the above-described embodimentsand a moving image decoding method complying with the MPEG4-AVCstandard. Examples of such partially common processes include entropyencoding, inverse quantization, deblocking filter, and motioncompensation. The common processes may be performed by commonly using adecoding processing unit ex902 that supports the MPEG4-AVC standard, andother processes that do not compatible with the MPEG4-AVC standard andthat are specific to an aspect of the present disclosure may beperformed using a dedicated decoding processing unit ex901. Inparticular, an aspect of the present disclosure focuses on motioncompensation. Thus, for example, the dedicated decoding processing unitex901 may be used for motion compensation, and a decoding processingunit may be commonly used for some or all of the other processes ofentropy decoding, deblocking filter, and inverse quantization. Anotherconfiguration may be used for sharing a decoding processing unit suchthat common processes are performed by commonly using a decodingprocessing unit for executing the moving image decoding methodillustrated in each of the above-described embodiments, and processesspecific to the MPEG4-AVC standard are performed using a dedicateddecoding processing unit.

FIG. 49B illustrates still another example ex1000 of sharing someprocesses. In the illustrated example, a dedicated decoding processingunit ex1001 that supports processes specific to an aspect of the presentdisclosure, a dedicated decoding processing unit ex1002 that supportsprocesses specific to other existing standards, and a shared decodingprocessing unit ex1003 that supports processes common to a moving imagedecoding method according to an aspect of the present disclosure andanother moving image decoding method conforming to an existing standardare used. The dedicated decoding processing units ex1001 and ex1002 maynot necessarily be dedicated to processes specific to an aspect of thepresent disclosure or processes specific to other existing standards,and may be configured to execute other general-purpose processingoperations. In addition, the configuration according to this embodimentmay also be implemented by the LSI ex500.

In the manner described above, sharing a decoding processing unit forprocesses common to a moving image decoding method according to anaspect of the present disclosure and a moving image decoding methodconforming to an existing standard can reduce the circuit scale of theLSI and also reduce cost.

The present disclosure is applicable to, for example, a TV receiver, adigital video recorder, a car navigation system, a mobile phone, adigital camera, a digital video camera, a content distribution server, acontent distribution system, and so forth.

What is claimed is:
 1. An image decoding apparatus comprising: aprocessor; and a non-transitory computer-readable recording mediumstoring thereon a computer program, which when executed by theprocessor, causes the processor to perform operations including:requesting an image transmission apparatus to transmit an image;transmitting range information to the image transmission apparatus, therange information indicating a range which is a partial range of theimage within an entire range of the image and which is selected inaccordance with a user operation; receiving a flag indicating whetherthe image belongs to a first type which is an advertisement or a secondtype which is not an advertisement; in a case that the partial range ofthe image is selected, (i) receiving the entirety of the image from theimage transmission apparatus when the flag indicates that the imagebelongs to the first type, and (ii) receiving one or more tilescorresponding to the partial range of the image among a plurality oftiles included in the image from the image transmission apparatus whenthe flag indicates that the image belongs to the second type, each ofthe one or more tiles being prohibited to refer to tiles outside of acorresponding tile of each of the one or more tiles in inter-pictureprediction; and decoding the entirety or the partial range of the image.2. The image decoding apparatus according to claim 1, the operationsfurther including: displaying the entirety or the decoded partial rangeof the image on a first display device.
 3. The image decoding apparatusaccording to claim 2, wherein the range information is stored in amemory of the image decoding apparatus, the receiving receives thepartial range of the image in accordance with the stored rangeinformation, the decoding decodes the partial range of the image inaccordance with the stored range information, and the displayingdisplays the partial range of the image in accordance with the storedrange information.
 4. The image decoding apparatus according to claim 3,wherein in a case where the displaying displays the partial range of theimage after displaying the entirety of another image, the displayingdisplays the partial range of the image in accordance with the storedrange information.
 5. The image decoding apparatus according to claim 4,wherein in a case where the displaying further stops displaying of thepartial range of the image and then resumes to display the partial rangeof the image, the displaying displays the partial range of the image inaccordance with the stored range information.
 6. The image decodingapparatus according to claim 5, wherein the displaying displays thepartial range of the image in accordance with the stored rangeinformation on a second display device different from the first displaydevice.
 7. The image decoding apparatus according to claim 6, wherein ina case where the range information is not stored, the transmittingtransmits the range information to the image transmission apparatus, andin a case where the range information is stored and the receivingreceives the partial range of the image, the receiving receives thepartial range of the image in accordance with the stored rangeinformation without the range information being transmitted to the imagetransmission apparatus.
 8. The image decoding apparatus according toclaim 7, the operations further including: selecting the partial rangeof the image in accordance with the user operation, wherein thetransmitting transmits the range information indicating the selectedpartial range of the image to the image transmission apparatus.
 9. Animage transmission apparatus comprising: a processor; a non-transitorycomputer-readable medium storing thereon a computer program, which whenexecuted by the processor, causes the processor to perform operationsincluding: accepting a request from an image decoding apparatus totransmit an image; receiving range information from the image decodingapparatus, the range information indicating a range which is a partialrange of the image within an entire range of the image and which isselected in accordance with a user operation; and in a case that thepartial range of the image is selected, (i) transmitting the entirety ofthe image and a flag indicating the image belonging to a first typewhich is an advertisement to the image decoding apparatus when the imagebelongs to the first type, and (ii) transmitting one or more tilescorresponding to the partial range of the image among a plurality oftiles included in the image and the flag indicating the image belongingto a second type which is not an advertisement to the image decodingapparatus when the image belongs to the second type, each of the one ormore tiles being prohibited to refer to tiles outside of a correspondingtile of each of the one or more tiles in inter-picture prediction. 10.An image processing system, comprising: an image transmission apparatus;and an image decoding apparatus, the image transmission apparatusincluding: a first processor; and a first non-transitorycomputer-readable recording medium storing thereon a first computerprogram, which when executed by the first processor, cause the firstprocessor to perform first operations including: transmitting an imageto the image decoding apparatus, the image decoding apparatus including:a second processor; and a second non-transitory computer-readablerecording medium storing thereon a second computer program, which whenexecuted by the second processor, cause the second processor to performsecond operations including: receiving the image from the imagetransmission apparatus and decoding the image received from the imagetransmission apparatus; requesting the image transmission apparatus totransmit the image, transmitting range information to the imagetransmission apparatus, the range information indicating a range whichis a partial range of the image within an entire range of the image andwhich is selected in accordance with a user operation; receiving a flagindicating whether the image belongs to a first type which is anadvertisement or a second type which is not an advertisement; in a casethat the partial range of the image is selected, (i) receiving theentirety of the image from the image transmission apparatus when theflag indicates that the image belongs to the first type, and (ii)receiving one or more tiles corresponding to the partial range of theimage among a plurality of tiles included in the image from the imagetransmission apparatus when the flag indicates that the image belongs tothe second type, each of the one or more tiles being prohibited to referto tiles outside of a corresponding tile of each of the one or moretiles in inter-picture prediction; and decoding the entirety or thepartial range of the image, the first operations of the imagetransmission apparatus further including: accepting a request from theimage decoding apparatus to transmit the image; receiving the rangeinformation from the image decoding apparatus; and in a case that thepartial range of the image is selected, (i) transmitting the entirety ofthe image and the flag indicating the image belonging to the first typeto the image decoding apparatus when the image belongs to the firsttype, and (ii) transmitting the one or more tiles corresponding to thepartial range of the image among the plurality of tiles included in theimage and the flag indicating the image belonging to the second type tothe image decoding apparatus when the image belongs to the second type.11. An image decoding method, comprising: requesting an imagetransmission apparatus to transmit an image; transmitting rangeinformation to the image transmission apparatus, the range informationindicating a range which is a partial range of the image within anentire range of the image and which is selected in accordance with auser operation; receiving a flag indicating whether the image belongs toa first type which is an advertisement or a second type which is not anadvertisement; in a case that the partial range of the image isselected, (i) receiving the entirety of the image from the imagetransmission apparatus when the flag indicates that the image belongs tothe first type, and (ii) receiving one or more tiles corresponding tothe partial range of the image among a plurality of tiles included inthe image from the image transmission apparatus when the flag indicatesthat the image belongs to the second type, each of the one or more tilesbeing prohibited to refer to tiles outside of a corresponding tile ofeach of the one or more tiles in inter-picture prediction; and decodingthe entirety or the partial range of the image.
 12. An imagetransmission method, comprising: accepting a request from an imagedecoding apparatus to transmit an image; receiving range informationfrom the image decoding apparatus, the range information indicating arange which is a partial range of the image within an entire range ofthe image and which is selected in accordance with a user operation; andin a case that the partial range of the image is selected, (i)transmitting the entirety of the image and a flag indicating the imagebelonging to a first type which is an advertisement to the imagedecoding apparatus when the image belongs to the first type, and (ii)transmitting one or more tiles corresponding to the partial range of theimage among a plurality of tiles included in the image and the flagindicating the image belonging to a second type which is not anadvertisement to the image decoding apparatus when the image belongs tothe second type, each of the one or more tiles being prohibited to referto tiles outside of a corresponding tile of each of the one or moretiles in inter-picture prediction.